-- Logs begin at Mon 2022-07-11 13:52:40 UTC, end at Mon 2022-07-11 15:00:17 UTC. -- Jul 11 14:24:48.891643 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 11 14:24:48.898998 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: *** Starting uWSGI 2.0.18-debian (64bit) on [Mon Jul 11 14:24:48 2022] *** Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: nodename: dv-cinder-master-3-iscsi Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: machine: x86_64 Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: clock source: unix Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: pcre jit disabled Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: detected number of CPU cores: 4 Jul 11 14:24:48.901042 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: current working directory: / Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: detected binary path: /usr/bin/uwsgi-core Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: your processes number limit is 47680 Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: your memory page size is 4096 bytes Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: detected max file descriptor number: 2048 Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: lock engine: pthread robust mutexes Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: thunder lock: enabled Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 11 14:24:48.901607 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] Jul 11 14:24:48.945886 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: Python main interpreter initialized at 0x5570603449c0 Jul 11 14:24:48.945886 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: python threads support enabled Jul 11 14:24:48.945886 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: your server socket listen backlog is limited to 100 connections Jul 11 14:24:48.945886 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: your mercy for graceful operations on workers is 90 seconds Jul 11 14:24:48.946234 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: mapped 403077 bytes (393 KB) for 2 cores Jul 11 14:24:48.946309 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: *** Operational MODE: preforking *** Jul 11 14:24:48.946576 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: *** uWSGI is running in multiple interpreter mode *** Jul 11 14:24:48.946576 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: spawned uWSGI master process (pid: 80414) Jul 11 14:24:48.946790 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 11 14:24:48.950539 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: spawned uWSGI worker 1 (pid: 80423, cores: 1) Jul 11 14:24:48.950539 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: spawned uWSGI worker 2 (pid: 80424, cores: 1) Jul 11 14:24:48.950539 dv-cinder-master-3-iscsi devstack@keystone.service[80414]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 11 14:24:54.456510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 11 14:24:54.456510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: from cryptography.utils import int_from_bytes Jul 11 14:24:54.457713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 11 14:24:54.457713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: from cryptography.utils import int_from_bytes Jul 11 14:24:54.507011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 11 14:24:54.507251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 11 14:24:54.507410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] command line args: [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 11 14:24:54.507698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 11 14:24:54.507899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ================================================================================ {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 11 14:24:54.508172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] admin_token = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.508579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] config_dir = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.508806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.508989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] config_source = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.509706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] debug = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.509908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.510420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.510552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.510735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.510981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.511211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] list_limit = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.511389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] log_config_append = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.511564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.511778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] log_dir = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.511905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] log_file = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.512141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.512388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.512642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.512875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.513050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.513225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.513465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.513705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.513937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.514168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.514400 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.514658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.514903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.515145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.515415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.515911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.72/identity {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.516093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] publish_errors = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.516345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.516589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.516772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.517017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.517251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.517484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] standard_threads = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.517712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.517951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.518126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.518300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] use_journal = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.518495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] use_json = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.518671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] use_stderr = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.518843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] use_syslog = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.519071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.519325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.519563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.519828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.520068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.520321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.520558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.520803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.521035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.521279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.521524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.521768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.521998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.522243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.522496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.522742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.522985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.523221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.523474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.523726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.523968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.524201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.524452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.525395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.525772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.526039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.526268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.526521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.526754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.526989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.527238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.527485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.527713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.527941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.528186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.528431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.528669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.528916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.529146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.529380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.529612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.529842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.530071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.530304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.530555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.530793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.531040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.531272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.531509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.531739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.531973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.532206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.532452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.532683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.532921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.533158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.533394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.533638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.533874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.534101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.534346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.534602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.534839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.535069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.535387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.535559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.535792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.536029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.536261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.536524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.536760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.537006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.537249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.537483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.537715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.537946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.538193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.538444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.538682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.538923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.539159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.539395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.539627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.539860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.540096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.540376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.540626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.540869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.541106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.541341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.541577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.541811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.542046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.542277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.542530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.542767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.543001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.543229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.543468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.543699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.543942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.544177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.544429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.544685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.544924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.545156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] auth.external = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.545386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.545678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.545909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.546139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] auth.password = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.546367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] auth.token = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.546633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.546865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.547107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.547336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.547579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.547826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.548054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.548287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] credential.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.548529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.548773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.549017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.549252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.549494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.549725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.549983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.550237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.550490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.550742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.551024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.551270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.551540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.551783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.552036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.552269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.552505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.552749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.552991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.553218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.553462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.553696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.553945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.554179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.554460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.554684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.554932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.555165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.555409 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.555652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.555884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.556313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.556770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.557202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.557644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.558100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.558598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.559021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.559446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.559836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.560117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.560360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.560627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.560893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.561147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.561423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.561831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.562290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.562764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.563150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.563467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.563777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.564091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.564510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.564941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.565348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.565786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.566219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.566662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.567095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.567542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.567861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.568171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.568510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.568961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.569407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.569862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.570311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.570668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.571010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.571276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.572207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.572450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.572742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.572998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.573240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.573480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.573720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.573953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.574478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.574748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.574981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.575300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.575475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.575720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.575955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.576188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.576419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.576652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.576889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.577120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.577364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.577615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.577861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.578104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.578347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.578598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.578848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.579088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.579327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.579564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.579799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.580045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.580300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.580538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.580792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.581022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.581269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.581505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.581729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.581962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.582205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.582503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.582736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.582975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.583217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.583466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.583711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.583947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.584194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.584445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.584680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.584924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.585159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.585397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.585630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] role.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.585874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] role.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.586103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.586352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.586624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.586870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.587117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.587411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.587660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.587904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.588147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.588379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.588622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.588866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.589110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.589353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.589782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.590018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.590270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.590546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.590794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.591052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.591289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.591545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.591796 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.592033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.592263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.592496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.592722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.592970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.593218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.593468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.593702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.593936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.594163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.594393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.594652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.594899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.595134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.595383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.595627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.595863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.596104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.596340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.596582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.596819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.597055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.597287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.597528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.597774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.598001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.598242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.598560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.598822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.connection = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.599063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.599680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.600001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.600146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.600382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.600616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.600852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.601096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.601333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.601574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.601806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.602052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.602287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.602553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.602784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.603538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.603770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.603940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 11 14:24:54.634000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 11 14:24:54.634000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: from cryptography.utils import int_from_bytes Jul 11 14:24:54.634965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 11 14:24:54.634965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: from cryptography.utils import int_from_bytes Jul 11 14:24:54.635247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.635480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.635704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.644249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.673334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.673334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.683157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 11 14:24:54.683427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 11 14:24:54.683548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] command line args: [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 11 14:24:54.683824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 11 14:24:54.684008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ================================================================================ {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 11 14:24:54.684265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] admin_token = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.684513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] config_dir = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.684704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.684889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] config_source = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.685074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] debug = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.685289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.685528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.685709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.685961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.686210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.686459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] list_limit = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.686645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] log_config_append = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.686826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.687008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] log_dir = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.687188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] log_file = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.687433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.687683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.688196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.688380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.688559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.688804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.689051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.689288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.689525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.689761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.689997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.690249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.690522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.690805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.691306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.72/identity {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.691487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] publish_errors = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.691723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.691954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.692139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.692391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.692629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.692863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] standard_threads = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.693096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.693337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.693516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.693700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] use_journal = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.693880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] use_json = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.694057 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] use_stderr = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.694236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] use_syslog = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.694488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 11 14:24:54.694745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.694985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.695248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.695492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.695742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.695984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.696238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.696476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.696752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.697001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.697258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.697492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.697787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.698026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.698275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.698552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.698797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.699053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.699563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.699812 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.700050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.700305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.700566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.700822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.701076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.701309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.701541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.701773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.702010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.702277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.702546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.702783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.703008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.703256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.703504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.703748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.703997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.704228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.704469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.704706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.704937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.705170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.705401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.705633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.705874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.706125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.706360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.706665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.706857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.707097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.707370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.707986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.707986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.707986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.707986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.709383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.709383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.709383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.709383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.709872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.709872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.710112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.710233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.710498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.710741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.710980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.711224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.711466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.711737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.711980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.712233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.712677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.712806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.712960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.713200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.713451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.713694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.713937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.714463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.714463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.714724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.714925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.715165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.715405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.715669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.715912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.716157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.716400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.716645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.716890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.717130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.717368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.717600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.717833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.718066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.718304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.718558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.718803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.719038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.719289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.719530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.719787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.720049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.720292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.720527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] auth.external = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.720761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.721055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.721292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.721525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] auth.password = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.721800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] auth.token = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.722005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.722240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.722516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.722751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.722999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.723250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.723485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.723723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] credential.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.724102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.724453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.724636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.724879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.725130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.725422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.725635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.725891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.726131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.726385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.726708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.726961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.727484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.727786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.728541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.728785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.729043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.729293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.729538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.729789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.730035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.730293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.730561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.730822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.731065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.731319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.731559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.731811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.732062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.732302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.732549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.732798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.733035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.733276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.733530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.733788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.734024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.734261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.734524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.734773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.735023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.735273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.735531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.735876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.736041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.736285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.736529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.736772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.737012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.737246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.737483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.737731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.737973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.738225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.738480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.738732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.738983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.739222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.739472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.739722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.739955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.740200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.740433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.740675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.740917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.741165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.741407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.741649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.741927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.742177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.742417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.742673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.742924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.743174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.743412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.743711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.743950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.744183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.744430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.744696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.744931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.745180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.745429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.745677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.745914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.746147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.746385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.746650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.746892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.747125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.747373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.747621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.747868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.748114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.748370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.748594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.748846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.749087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.749327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.749568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.749806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.750054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.750303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.750569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.750826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.751061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.751323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.751565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.751802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.752044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.752295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.752544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.752779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.753022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.753268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.753521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.753770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.754011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.754261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.754555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.754796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.755045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.755286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.755529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.755766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] role.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.756731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] role.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.756731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.756731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.756731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.757238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.757238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.757472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.757674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.757922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.758187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.758415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.758688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.758935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.759187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.759962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.760408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.760651 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.760908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.761161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.761417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.761674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.761913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.762191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.762558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.763939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.763939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.763939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.763939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.763939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.763939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.763939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.767111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.771408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.connection = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.772413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.773300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.774004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.774004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.774004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.774004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 11 14:24:54.774004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 11 14:24:54.801787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.802196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.802319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.811846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.828581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.830679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.830916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.878669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.878882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.949608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.949839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.950064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.950373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.950678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.950968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.955518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.976528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.976689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.976897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.977114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.977326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.977534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.977743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.988854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.991931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:54.992175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.002173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.002403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.002665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.003100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.003312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.003561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.021852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.022056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 11 14:24:55.022390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.022620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 11 14:24:55.022887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.023144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.023399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.023652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.034544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.034826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.035087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.035375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.037223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.037457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.037683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.037903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.038123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.038342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.043729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.047049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.047330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.047613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.047892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.048168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.048423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.048687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.048954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.049219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.049485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.049748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.052891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.053125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.053354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.053585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.053805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.054029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.054250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.062533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.065665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.065992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.066131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.066334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.066590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.066809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.067019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 11 14:24:55.078088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.078485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.078700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 11 14:24:55.079043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.079247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 11 14:24:55.079518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.079756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.079840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.080107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.080283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.085293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.085572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.085975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.091519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.091802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.092074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.092365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.093344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.093627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.098171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.098598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.098768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.098967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.099232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.099500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.099762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.100026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.100679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.104106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.104409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.104697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.104979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.105258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.105516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.105784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.106171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.106478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.106773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.107045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.119256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.119534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.124169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.127819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.128109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.133249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.135085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.136887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.137177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.137442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.141141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.141447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.141756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.142034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.142310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.142563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.142650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.142848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.143130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.150695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.150981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.155562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.155973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.156257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.156551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.156835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.157115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.157383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.157663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.162243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.162565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.162832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.163101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.171100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.172800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.173069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.176280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.176605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.176726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.176900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.177009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.177205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.177546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.182994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.186748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.187045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.192116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.192455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.192735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.193013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.193304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.193584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.195335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.195626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.195767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.200409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.200706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.200996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.201276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.201551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.201828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.214673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.217959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.220977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.221268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.221351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.221538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.221801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.223014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.224651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.224924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.225206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.230481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.232129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.232405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.233085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.234726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.235501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.235781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.236070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.236352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.236627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.237958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.238247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.238664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.238950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.248619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.248904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.249178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.250752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.251034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.251317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.251609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.251890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.252175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.252453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.257058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.257331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.257605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.257877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.258147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.258559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.258834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.259126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.259406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.259676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.259944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.260216 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.260479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.260743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.272022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.275443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.278816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.280492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.282147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.282472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.282750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.290265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.292470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.295777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.296050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.296380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.296620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.299060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WSGI app 0 (mountpoint='') ready in 7 seconds on interpreter 0x5570603449c0 pid: 80424 (default app) Jul 11 14:24:55.302632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f893e1-76dc-4c90-b422-2006f82bdfa9 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:24:55.302927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f893e1-76dc-4c90-b422-2006f82bdfa9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:24:55.303204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f893e1-76dc-4c90-b422-2006f82bdfa9 None None] PATH_INFO: `/v3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:24:55.305913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.306335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.306893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.307757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1/1] 10.222.0.72 () {50 vars in 791 bytes} [Mon Jul 11 14:24:55 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:24:55.315165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 11 14:24:55.315453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.315735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.316013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.316292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.316622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.316988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.317260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.317541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.317811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.318084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.318354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.318640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.318908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 11 14:24:55.358794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WSGI app 0 (mountpoint='') ready in 7 seconds on interpreter 0x5570603449c0 pid: 80423 (default app) Jul 11 14:25:03.035916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f842d8-6456-44e5-b51d-b916668f53ec None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:03.036515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f842d8-6456-44e5-b51d-b916668f53ec None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:03.036515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f842d8-6456-44e5-b51d-b916668f53ec None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:03.037815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 2/2] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:03.045424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:03.045947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:03.046040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:03.115583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: INFO dbcounter [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] Registered counter for database keystone Jul 11 14:25:03.138264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG oslo_db.sqlalchemy.engines [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80423) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 11 14:25:03.144701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writer thread running {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 11 14:25:03.333754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG passlib.handlers.bcrypt [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80423) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 11 14:25:03.334089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG passlib.handlers.bcrypt [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80423) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 11 14:25:03.390214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:03.391883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:03.525251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1/3] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:03 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 483 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 11 14:25:03.702974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f842d8-6456-44e5-b51d-b916668f53ec None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:03.773877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: INFO dbcounter [None req-44f842d8-6456-44e5-b51d-b916668f53ec None None] Registered counter for database keystone Jul 11 14:25:03.793999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG oslo_db.sqlalchemy.engines [None req-44f842d8-6456-44e5-b51d-b916668f53ec None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80424) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 11 14:25:03.800422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writer thread running {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 11 14:25:03.921828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f842d8-6456-44e5-b51d-b916668f53ec None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:03.926230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc833ec-8ebd-4be1-a876-055a803efae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:03.927132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc833ec-8ebd-4be1-a876-055a803efae7 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:03.927420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc833ec-8ebd-4be1-a876-055a803efae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:03.927697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc833ec-8ebd-4be1-a876-055a803efae7 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:03.937228 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc833ec-8ebd-4be1-a876-055a803efae7 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:03.990551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING keystone.common.rbac_enforcer.enforcer [None req-8dc833ec-8ebd-4be1-a876-055a803efae7 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 11 14:25:03.990891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc833ec-8ebd-4be1-a876-055a803efae7 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:04.001443 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 3/4] 10.222.0.72 () {58 vars in 1324 bytes} [Mon Jul 11 14:25:03 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 11 14:25:04.007881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:04.026498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc852465-1a1e-415b-b49c-64761e8545a7 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:04.031177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e77802d-896d-40b2-a943-6647b827b6cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:04.032314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77802d-896d-40b2-a943-6647b827b6cc admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:04.032600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77802d-896d-40b2-a943-6647b827b6cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:04.032882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77802d-896d-40b2-a943-6647b827b6cc admin admin] PATH_INFO: `/v3/projects/e2b4fae2f8fe4791bd2046666efe9390` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:04.034414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e77802d-896d-40b2-a943-6647b827b6cc admin admin] RBAC: Authorizing `identity:get_project(project_id=e2b4fae2f8fe4791bd2046666efe9390)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:04.085461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING keystone.common.rbac_enforcer.enforcer [None req-6e77802d-896d-40b2-a943-6647b827b6cc admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 11 14:25:04.086524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e77802d-896d-40b2-a943-6647b827b6cc admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:04.093568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 2/5] 10.222.0.72 () {56 vars in 1207 bytes} [Mon Jul 11 14:25:04 2022] GET /identity/v3/projects/e2b4fae2f8fe4791bd2046666efe9390 => generated 344 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:06.142514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bd4e2-a0e7-4f21-9e5c-4bbe4e678103 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:06.143069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bd4e2-a0e7-4f21-9e5c-4bbe4e678103 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:06.143069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bd4e2-a0e7-4f21-9e5c-4bbe4e678103 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:06.147197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 4/6] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:06.153796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a3fb2-2555-441c-a4f0-3ba8377202a0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:06.154263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a3fb2-2555-441c-a4f0-3ba8377202a0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:06.154359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a3fb2-2555-441c-a4f0-3ba8377202a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:06.173782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e17a3fb2-2555-441c-a4f0-3ba8377202a0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:06.175368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e17a3fb2-2555-441c-a4f0-3ba8377202a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:06.183331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 3/7] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:06 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 11 14:25:06.206015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569bd4e2-a0e7-4f21-9e5c-4bbe4e678103 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:06.218914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569bd4e2-a0e7-4f21-9e5c-4bbe4e678103 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:06.223182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d56783-27ca-4180-9af4-bd8bacba88ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:06.223832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d56783-27ca-4180-9af4-bd8bacba88ce admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:06.224122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d56783-27ca-4180-9af4-bd8bacba88ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:06.224407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d56783-27ca-4180-9af4-bd8bacba88ce admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:06.233507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d56783-27ca-4180-9af4-bd8bacba88ce admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:06.235356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d56783-27ca-4180-9af4-bd8bacba88ce admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:06.244699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 5/8] 10.222.0.72 () {58 vars in 1324 bytes} [Mon Jul 11 14:25:06 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 11 14:25:06.250517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a3fb2-2555-441c-a4f0-3ba8377202a0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:06.260061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a3fb2-2555-441c-a4f0-3ba8377202a0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:06.264541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d25e841-840a-48d3-8a7c-9981f9f1a2c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:06.265414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d25e841-840a-48d3-8a7c-9981f9f1a2c9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:06.265695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d25e841-840a-48d3-8a7c-9981f9f1a2c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:06.265968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d25e841-840a-48d3-8a7c-9981f9f1a2c9 admin admin] PATH_INFO: `/v3/users/a7d8b4db44114e1ba544d78dc12ef30a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:06.267383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d25e841-840a-48d3-8a7c-9981f9f1a2c9 admin admin] RBAC: Authorizing `identity:get_user(user_id=a7d8b4db44114e1ba544d78dc12ef30a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:06.269263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d25e841-840a-48d3-8a7c-9981f9f1a2c9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:06.270968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 4/9] 10.222.0.72 () {56 vars in 1198 bytes} [Mon Jul 11 14:25:06 2022] GET /identity/v3/users/a7d8b4db44114e1ba544d78dc12ef30a => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:08.416232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa69070-5dab-46c5-b3fc-e62ba6a1b5bc None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:08.416748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa69070-5dab-46c5-b3fc-e62ba6a1b5bc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:08.416831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa69070-5dab-46c5-b3fc-e62ba6a1b5bc None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:08.418177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 6/10] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:08.424653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5db02-028d-4be8-8266-5506fe26d410 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:08.425062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5db02-028d-4be8-8266-5506fe26d410 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:08.425306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5db02-028d-4be8-8266-5506fe26d410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:08.444080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-5fa5db02-028d-4be8-8266-5506fe26d410 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:08.445267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-5fa5db02-028d-4be8-8266-5506fe26d410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:08.448380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaa0d31-44c5-4011-8e2b-53d51f6a49b6 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:08.448702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaa0d31-44c5-4011-8e2b-53d51f6a49b6 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:08.448970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaa0d31-44c5-4011-8e2b-53d51f6a49b6 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:08.450622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 7/11] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:08.456284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:08.456665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:08.456972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:08.498627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG passlib.handlers.bcrypt [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80424) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 11 14:25:08.499052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG passlib.handlers.bcrypt [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80424) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 11 14:25:08.535745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 5/12] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:08.545738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:08.547421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:08.558693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa5db02-028d-4be8-8266-5506fe26d410 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:08.585437 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:08.658915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 8/13] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:08.675529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa5db02-028d-4be8-8266-5506fe26d410 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:08.685303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:08.712905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:08.713565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:08.713850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:08.714130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:08.784524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:08.786037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:08.798527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772a702d-e500-4aab-96ea-90d1ee87bda7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:08.801139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:08.810996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:08.841825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:08.842924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:08.843215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:08.843497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:08.855958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:08.867530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:08.868561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:08.869522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 9/14] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:08.874816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:08.900148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 6/15] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 11 14:25:08.906724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:08.982301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb19080-19cb-47c1-8679-c7fdd9cee006 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:09.018645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:09.019327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:09.019605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:09.019909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:09.020642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:09.020952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:09.022484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:09.023184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6326c900-4f35-4031-8f56-6dc4307e6c55 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:09.035508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 10/16] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:09.041099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:09.052753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:09.066758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:09.067581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:09.067878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:09.068164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:09.075582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:09.081215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:09.082917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:25:09.105518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:09.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:09.107307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:25:09.107307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:25:09.107307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:25:09.107307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:09.107307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 7/17] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:09.112725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:09.155032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3f8c2-a747-412b-a74f-1ed71c7a4124 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:09.191074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:09.191834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:09.192224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:09.192401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] PATH_INFO: `/v3/users/a7d8b4db44114e1ba544d78dc12ef30a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:09.194055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] RBAC: Authorizing `identity:get_user(user_id=a7d8b4db44114e1ba544d78dc12ef30a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:09.195578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:09.197819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 11/18] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:09 2022] GET /identity/v3/users/a7d8b4db44114e1ba544d78dc12ef30a => generated 247 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:09.204365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:09.221337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1e5f42-6487-4cf6-ab0a-6ca6cc8d36bc None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:09.260592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:09.261271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:09.261601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:09.261847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:09.262619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:09.262983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:09.264378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:09.283000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 8/19] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:25:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:09.317864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:09.327539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414de470-df39-46d4-994d-6bbd7ced8cab None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:09.370524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3289859-ad2e-4c1d-8b11-045f9142229a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:09.371429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3289859-ad2e-4c1d-8b11-045f9142229a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:09.371715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3289859-ad2e-4c1d-8b11-045f9142229a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:09.372031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3289859-ad2e-4c1d-8b11-045f9142229a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:09.373463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3289859-ad2e-4c1d-8b11-045f9142229a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:09.375049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3289859-ad2e-4c1d-8b11-045f9142229a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:09.377581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 12/20] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:09.385639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:09.426454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:09.502041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b01eba-cd14-43f1-b230-d0ab8d2fbb9b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:09.544980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56924f89-e693-44fd-aa57-3a26282ab9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:09.545590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56924f89-e693-44fd-aa57-3a26282ab9b7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:09.545933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56924f89-e693-44fd-aa57-3a26282ab9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:09.546217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56924f89-e693-44fd-aa57-3a26282ab9b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:09.547167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56924f89-e693-44fd-aa57-3a26282ab9b7 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.domain.id=default, user.id=a7d8b4db44114e1ba544d78dc12ef30a) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:09.547490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56924f89-e693-44fd-aa57-3a26282ab9b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:09.548966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56924f89-e693-44fd-aa57-3a26282ab9b7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:09.558873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 9/21] 10.222.0.72 () {56 vars in 1322 bytes} [Mon Jul 11 14:25:09 2022] GET /identity/v3/role_assignments?user.id=a7d8b4db44114e1ba544d78dc12ef30a&scope.domain.id=default&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 233 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:12.104217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d8e71a-3f60-4bb3-98f7-a5d97a9aa8eb None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.104839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d8e71a-3f60-4bb3-98f7-a5d97a9aa8eb None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.104839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d8e71a-3f60-4bb3-98f7-a5d97a9aa8eb None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.106177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 13/22] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:12.107784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84862a26-3182-4fa3-8c56-89de3e4089db None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.108194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84862a26-3182-4fa3-8c56-89de3e4089db None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.108362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84862a26-3182-4fa3-8c56-89de3e4089db None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.109711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 10/23] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:12.112058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086aad7-842e-419c-923d-2b3448e5af97 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.112354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086aad7-842e-419c-923d-2b3448e5af97 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.112636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086aad7-842e-419c-923d-2b3448e5af97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.123120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18dce39-d8b5-4f4d-9bab-578cacd66f7e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.123554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18dce39-d8b5-4f4d-9bab-578cacd66f7e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.123948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18dce39-d8b5-4f4d-9bab-578cacd66f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.141396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f086aad7-842e-419c-923d-2b3448e5af97 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:12.147568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f086aad7-842e-419c-923d-2b3448e5af97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:12.172038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e18dce39-d8b5-4f4d-9bab-578cacd66f7e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:12.173253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e18dce39-d8b5-4f4d-9bab-578cacd66f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:12.285132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 14/24] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:12.307789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfeb966-e9ce-4c1f-811b-7acdc863236b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.308107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfeb966-e9ce-4c1f-811b-7acdc863236b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.308361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfeb966-e9ce-4c1f-811b-7acdc863236b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.309787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 15/25] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:12.316037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0caa5c-7b50-417e-9fd9-504f5869f724 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.316378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0caa5c-7b50-417e-9fd9-504f5869f724 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.316608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0caa5c-7b50-417e-9fd9-504f5869f724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.324720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 11/26] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:12.325971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18dce39-d8b5-4f4d-9bab-578cacd66f7e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:12.347497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1e0caa5c-7b50-417e-9fd9-504f5869f724 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:12.348738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1e0caa5c-7b50-417e-9fd9-504f5869f724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:12.366939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:12.480904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 16/27] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:12.482304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0caa5c-7b50-417e-9fd9-504f5869f724 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:12.497788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18dce39-d8b5-4f4d-9bab-578cacd66f7e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:12.518458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:12.553781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:12.554653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.554944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.555236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.588687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:12.598798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:12.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:12.600464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:12.603700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 12/28] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:12.604777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:12.655524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0caa5c-7b50-417e-9fd9-504f5869f724 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:12.699185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b6cc70-1ba7-4325-83df-db1223e315cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:12.700386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6cc70-1ba7-4325-83df-db1223e315cb None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.700778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6cc70-1ba7-4325-83df-db1223e315cb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.701158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6cc70-1ba7-4325-83df-db1223e315cb None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.702095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b6cc70-1ba7-4325-83df-db1223e315cb None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:12.703822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b6cc70-1ba7-4325-83df-db1223e315cb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:12.735007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:12.752647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5714e840-67ed-4b37-a1a2-f0e188a6b88e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:12.800179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dfe2d4-de2e-4318-bdb4-873b7f0aa534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:12.800871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dfe2d4-de2e-4318-bdb4-873b7f0aa534 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.801158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dfe2d4-de2e-4318-bdb4-873b7f0aa534 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.801441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dfe2d4-de2e-4318-bdb4-873b7f0aa534 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.802203 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dfe2d4-de2e-4318-bdb4-873b7f0aa534 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:12.803668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dfe2d4-de2e-4318-bdb4-873b7f0aa534 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:12.847506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 17/29] 10.222.0.72 () {60 vars in 1131 bytes} [Mon Jul 11 14:25:12 2022] POST /identity/v3/roles => generated 219 bytes in 365 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:12.848933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17abf39-a1b0-4dc0-a23f-24edce8880ca None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.849310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17abf39-a1b0-4dc0-a23f-24edce8880ca None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.849661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17abf39-a1b0-4dc0-a23f-24edce8880ca None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.851059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 18/30] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:12.851883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17abf39-a1b0-4dc0-a23f-24edce8880ca None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:12.945613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 13/31] 10.222.0.72 () {60 vars in 1131 bytes} [Mon Jul 11 14:25:12 2022] POST /identity/v3/roles => generated 225 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:12.947156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22870071-fde6-486a-a32d-3d7ce6608881 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:12.947461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22870071-fde6-486a-a32d-3d7ce6608881 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:12.947749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22870071-fde6-486a-a32d-3d7ce6608881 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:12.968872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-22870071-fde6-486a-a32d-3d7ce6608881 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:12.970636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-22870071-fde6-486a-a32d-3d7ce6608881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:12.973769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:13.002599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17abf39-a1b0-4dc0-a23f-24edce8880ca None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:13.049287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:13.049991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.050299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.050893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.051765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:13.052084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:13.053892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:13.065010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 19/32] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:13.071316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:13.079083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 14/33] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:13.104175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22870071-fde6-486a-a32d-3d7ce6608881 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:13.193722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9b3993-66d3-46a8-8035-2b8c2dc6a385 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:13.200384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:13.227986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22870071-fde6-486a-a32d-3d7ce6608881 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:13.238724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:13.240079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.240511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.241159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] PATH_INFO: `/v3/users/a7d8b4db44114e1ba544d78dc12ef30a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.242716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] RBAC: Authorizing `identity:get_user(user_id=a7d8b4db44114e1ba544d78dc12ef30a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:13.244532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:13.246452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 20/34] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:13 2022] GET /identity/v3/users/a7d8b4db44114e1ba544d78dc12ef30a => generated 247 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:13.255563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:13.275160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:13.276105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.276408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.276784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.281503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:13.295634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:13.299307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:25:13.322477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:13.323511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:13.324759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:25:13.324759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:25:13.324759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:25:13.324759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:13.328664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 15/35] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:13.331058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:13.441185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59977609-7da6-4e4a-997e-e97528890995 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:13.456122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff4b5dd-cf4e-44a8-9823-e96e173f3e0a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:13.489149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:13.499427 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:13.500058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:13.500686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.500790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.500995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.501089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.501305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.501409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.502105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:13.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:13.503158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:13.503945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:13.504712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:13.507308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 21/36] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:13.515681 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:13.519574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 16/37] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:25:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:13.526111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:13.634846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:13.645782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6910fe-8478-48b6-9638-e8bb72b17b56 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:13.661062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a502581-29a6-4d34-86f1-0c9cf29fc8d5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:13.692480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c0e234-eb59-465d-bb4a-439041c8c171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:13.693279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c0e234-eb59-465d-bb4a-439041c8c171 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.693583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c0e234-eb59-465d-bb4a-439041c8c171 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.693853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c0e234-eb59-465d-bb4a-439041c8c171 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.694641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c0e234-eb59-465d-bb4a-439041c8c171 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:13.696396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c0e234-eb59-465d-bb4a-439041c8c171 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:13.706948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bb23ba-1c26-4668-8296-6d25a46f2c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:13.707779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bb23ba-1c26-4668-8296-6d25a46f2c89 None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.708101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bb23ba-1c26-4668-8296-6d25a46f2c89 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.708525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bb23ba-1c26-4668-8296-6d25a46f2c89 None admin] PATH_INFO: `/v3/domains/default/users/a7d8b4db44114e1ba544d78dc12ef30a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.710829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bb23ba-1c26-4668-8296-6d25a46f2c89 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=a7d8b4db44114e1ba544d78dc12ef30a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:13.713179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bb23ba-1c26-4668-8296-6d25a46f2c89 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:13.729642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 22/38] 10.222.0.72 () {58 vars in 1361 bytes} [Mon Jul 11 14:25:13 2022] PUT /identity/v3/domains/default/users/a7d8b4db44114e1ba544d78dc12ef30a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:25:13.738710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 17/39] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:25:13 2022] POST /identity/v3/projects => generated 301 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:13.852385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef9a48f-55eb-45db-8710-c64a828504c5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.853273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef9a48f-55eb-45db-8710-c64a828504c5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.853500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef9a48f-55eb-45db-8710-c64a828504c5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:13.855014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 23/40] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:13.864433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bb6995-6f1a-4834-99c0-f16516746c17 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:13.865173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bb6995-6f1a-4834-99c0-f16516746c17 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:13.865795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bb6995-6f1a-4834-99c0-f16516746c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:14.009157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-56bb6995-6f1a-4834-99c0-f16516746c17 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:14.010842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-56bb6995-6f1a-4834-99c0-f16516746c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:14.057636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:14.073575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a1924a-02e2-45a7-af5b-c56f818a88c3 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:14.073865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a1924a-02e2-45a7-af5b-c56f818a88c3 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:14.074154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a1924a-02e2-45a7-af5b-c56f818a88c3 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:14.075952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 24/41] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:14.082678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab0e7e0-bb11-4a3b-aeef-ab7a36f917b7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:14.082678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab0e7e0-bb11-4a3b-aeef-ab7a36f917b7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:14.082678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab0e7e0-bb11-4a3b-aeef-ab7a36f917b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:14.101722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-6ab0e7e0-bb11-4a3b-aeef-ab7a36f917b7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:14.104001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6ab0e7e0-bb11-4a3b-aeef-ab7a36f917b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:14.113028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 18/42] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:14.137701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bb6995-6f1a-4834-99c0-f16516746c17 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:14.199830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 25/43] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:14.225196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab0e7e0-bb11-4a3b-aeef-ab7a36f917b7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:14.270990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bb6995-6f1a-4834-99c0-f16516746c17 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:14.313462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:14.314280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:14.314615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:14.315045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:14.316486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:14.317956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:14.319806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 19/44] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:14.320098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:14.326770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:14.366930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab0e7e0-bb11-4a3b-aeef-ab7a36f917b7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:14.388364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:14.410242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e907507e-7d62-4bbf-8f4d-dab9e64a1cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:14.411561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907507e-7d62-4bbf-8f4d-dab9e64a1cb2 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:14.411985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907507e-7d62-4bbf-8f4d-dab9e64a1cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:14.412473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907507e-7d62-4bbf-8f4d-dab9e64a1cb2 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:14.413648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e907507e-7d62-4bbf-8f4d-dab9e64a1cb2 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:14.415714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e907507e-7d62-4bbf-8f4d-dab9e64a1cb2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:14.454818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 26/45] 10.222.0.72 () {60 vars in 1131 bytes} [Mon Jul 11 14:25:14 2022] POST /identity/v3/roles => generated 223 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:25:14.470678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce5cd3a-9551-4b0b-95b7-0e13dba4468c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:14.518781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53959337-cbe4-41b2-bbb3-aad7f0693da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:14.519858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53959337-cbe4-41b2-bbb3-aad7f0693da4 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:14.520327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53959337-cbe4-41b2-bbb3-aad7f0693da4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:14.520786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53959337-cbe4-41b2-bbb3-aad7f0693da4 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:14.521780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53959337-cbe4-41b2-bbb3-aad7f0693da4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:14.523630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53959337-cbe4-41b2-bbb3-aad7f0693da4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:14.556412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 20/46] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:25:14 2022] POST /identity/v3/projects => generated 312 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:15.914450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1a15a6-83f3-4b61-aa6b-d546c29564db None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:15.914450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1a15a6-83f3-4b61-aa6b-d546c29564db None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:15.914450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1a15a6-83f3-4b61-aa6b-d546c29564db None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:15.914450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 27/47] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:15.920545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17496a9e-b096-4903-af4a-d6c6c668da5d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:15.920545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17496a9e-b096-4903-af4a-d6c6c668da5d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:15.920545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17496a9e-b096-4903-af4a-d6c6c668da5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:15.942920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-17496a9e-b096-4903-af4a-d6c6c668da5d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:15.945060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-17496a9e-b096-4903-af4a-d6c6c668da5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:16.027466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 21/48] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:16.051723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1a15a6-83f3-4b61-aa6b-d546c29564db None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:16.193487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1a15a6-83f3-4b61-aa6b-d546c29564db None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:16.245372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:16.261380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:16.263099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:16.263966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:16.264576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:16.281353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:16.293857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:16.294843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:16.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 28/49] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:16.302258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17496a9e-b096-4903-af4a-d6c6c668da5d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:16.333009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:16.434140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17496a9e-b096-4903-af4a-d6c6c668da5d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:16.477316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:16.478033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:16.478329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:16.478658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:16.479434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:16.479751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:16.481181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:16.494711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 22/50] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:16.500298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:16.625577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f7d240-6341-4ce1-93cf-582ff4e819f5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:16.664137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01feb43b-3e4f-4268-b510-ede6372c7eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:16.665572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01feb43b-3e4f-4268-b510-ede6372c7eea None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:16.665856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01feb43b-3e4f-4268-b510-ede6372c7eea None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:16.666136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01feb43b-3e4f-4268-b510-ede6372c7eea None admin] PATH_INFO: `/v3/users/a7d8b4db44114e1ba544d78dc12ef30a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:16.667434 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01feb43b-3e4f-4268-b510-ede6372c7eea None admin] RBAC: Authorizing `identity:get_user(user_id=a7d8b4db44114e1ba544d78dc12ef30a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:16.671053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01feb43b-3e4f-4268-b510-ede6372c7eea None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:16.671053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 29/51] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:16 2022] GET /identity/v3/users/a7d8b4db44114e1ba544d78dc12ef30a => generated 247 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:16.676170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:16.757379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d14b5-b2b4-4db4-a708-60a4b15dd1fd None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:16.757679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d14b5-b2b4-4db4-a708-60a4b15dd1fd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:16.758103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d14b5-b2b4-4db4-a708-60a4b15dd1fd None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:16.759508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 30/52] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:16.764728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e67003-973a-4d43-b7e0-74f6ec6a92b7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:16.765022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e67003-973a-4d43-b7e0-74f6ec6a92b7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:16.765307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e67003-973a-4d43-b7e0-74f6ec6a92b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:16.772420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:16.784767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d8e67003-973a-4d43-b7e0-74f6ec6a92b7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:16.786188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d8e67003-973a-4d43-b7e0-74f6ec6a92b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:16.793498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d46a10-11a2-47ef-98c6-3941877744ad None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:16.831618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:16.832478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:16.832903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:16.833196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:16.834462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:16.835977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:16.837900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 23/53] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:16.846476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:16.851781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:16.869320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 31/54] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:16.893071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e67003-973a-4d43-b7e0-74f6ec6a92b7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:16.972089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c42486a-e5c8-45df-acbd-aa2dace0bd87 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:17.013119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:17.013721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:17.014012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:17.014298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:17.015209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.domain.id=default, user.id=a7d8b4db44114e1ba544d78dc12ef30a) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:17.015536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:17.017054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:17.024873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 24/55] 10.222.0.72 () {56 vars in 1322 bytes} [Mon Jul 11 14:25:16 2022] GET /identity/v3/role_assignments?user.id=a7d8b4db44114e1ba544d78dc12ef30a&scope.domain.id=default&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 530 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:17.120282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e67003-973a-4d43-b7e0-74f6ec6a92b7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:17.163820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665bc59-c0a1-46fe-ba76-9bde0406395e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:17.164642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665bc59-c0a1-46fe-ba76-9bde0406395e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:17.164960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665bc59-c0a1-46fe-ba76-9bde0406395e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:17.165237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665bc59-c0a1-46fe-ba76-9bde0406395e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:17.166541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665bc59-c0a1-46fe-ba76-9bde0406395e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:17.168059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665bc59-c0a1-46fe-ba76-9bde0406395e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:17.169789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 32/56] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:17.175760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:17.197914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:17.293205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6732063d-fddb-4b3f-9b62-17236c062c02 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:17.332131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e25faf-9cf3-47c0-8ba0-4f22e695257e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:17.332797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e25faf-9cf3-47c0-8ba0-4f22e695257e None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:17.333087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e25faf-9cf3-47c0-8ba0-4f22e695257e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:17.333367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e25faf-9cf3-47c0-8ba0-4f22e695257e None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:17.334091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e25faf-9cf3-47c0-8ba0-4f22e695257e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:17.335583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e25faf-9cf3-47c0-8ba0-4f22e695257e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:17.364746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 25/57] 10.222.0.72 () {60 vars in 1140 bytes} [Mon Jul 11 14:25:17 2022] POST /identity/v3/projects => generated 298 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:19.476481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54012f14-3f29-4193-a876-5f1f4af1be58 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:19.477024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54012f14-3f29-4193-a876-5f1f4af1be58 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:19.477146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54012f14-3f29-4193-a876-5f1f4af1be58 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:19.478467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 33/58] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:19.485193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b1c7b-be61-4150-94d4-3ef0613a62cd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:19.485704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b1c7b-be61-4150-94d4-3ef0613a62cd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:19.486173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b1c7b-be61-4150-94d4-3ef0613a62cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:19.519335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ae9b1c7b-be61-4150-94d4-3ef0613a62cd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:19.520712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ae9b1c7b-be61-4150-94d4-3ef0613a62cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:19.585634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:19.594128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 26/59] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:19.619321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54012f14-3f29-4193-a876-5f1f4af1be58 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:19.681352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:19.728217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54012f14-3f29-4193-a876-5f1f4af1be58 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:19.762202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248b9319-2b76-4600-a265-0853f14fd46e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:19.763031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b9319-2b76-4600-a265-0853f14fd46e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:19.763318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b9319-2b76-4600-a265-0853f14fd46e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:19.763601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b9319-2b76-4600-a265-0853f14fd46e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:19.764791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248b9319-2b76-4600-a265-0853f14fd46e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:19.766263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248b9319-2b76-4600-a265-0853f14fd46e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:19.767918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 34/60] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:19.773567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9b1c7b-be61-4150-94d4-3ef0613a62cd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:19.873881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9b1c7b-be61-4150-94d4-3ef0613a62cd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:19.910548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0a8dae-76e6-4cf0-9554-ab5cbebb4518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:19.911232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0a8dae-76e6-4cf0-9554-ab5cbebb4518 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:19.911653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0a8dae-76e6-4cf0-9554-ab5cbebb4518 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:19.911807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0a8dae-76e6-4cf0-9554-ab5cbebb4518 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:19.912546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0a8dae-76e6-4cf0-9554-ab5cbebb4518 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:19.914029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0a8dae-76e6-4cf0-9554-ab5cbebb4518 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:19.945583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 27/61] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:25:19 2022] POST /identity/v3/users => generated 275 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:25:23.382206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bad567-247c-4c5b-9007-24c561c0d824 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:23.383396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bad567-247c-4c5b-9007-24c561c0d824 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:23.383902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bad567-247c-4c5b-9007-24c561c0d824 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:23.385532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 35/62] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:23.391233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab6743-a4d0-4f1e-b455-9ade2ce7765b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:23.392147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab6743-a4d0-4f1e-b455-9ade2ce7765b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:23.392695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab6743-a4d0-4f1e-b455-9ade2ce7765b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:23.419636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-32ab6743-a4d0-4f1e-b455-9ade2ce7765b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:23.421250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-32ab6743-a4d0-4f1e-b455-9ade2ce7765b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:23.490934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:23.520008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 28/63] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:23.523209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa59f2fd-cde1-4290-8a7d-9eb4ea954b1f None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:23.523500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa59f2fd-cde1-4290-8a7d-9eb4ea954b1f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:23.523781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa59f2fd-cde1-4290-8a7d-9eb4ea954b1f None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:23.525801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 36/64] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:23.532925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405d187-55db-4007-8c3b-d3f638b7c79b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:23.533242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405d187-55db-4007-8c3b-d3f638b7c79b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:23.533536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405d187-55db-4007-8c3b-d3f638b7c79b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:23.548791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa59f2fd-cde1-4290-8a7d-9eb4ea954b1f None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:23.559214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-0405d187-55db-4007-8c3b-d3f638b7c79b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:23.560544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0405d187-55db-4007-8c3b-d3f638b7c79b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:23.640352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 29/65] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:23.674460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0405d187-55db-4007-8c3b-d3f638b7c79b None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:23.756499 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa59f2fd-cde1-4290-8a7d-9eb4ea954b1f None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:23.783476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0405d187-55db-4007-8c3b-d3f638b7c79b None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:23.797637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:23.808217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:23.825161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:23.825371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:23.826054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:23.826346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:23.826654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:23.826789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:23.827286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:23.827706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:23.841605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:23.848393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:23.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:23.851392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:23.852355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:23.858764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 30/66] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:23.860892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:23.861887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:23.862938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:23.863790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 37/67] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:23.870037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:23.984384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b1d918-8d3d-4120-a9b0-a073ce31207a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.004160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5885e8-367a-4af2-b0b4-cdfb46472240 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.029149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412b5350-2934-41d2-b82e-5fa4a13e80a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.029847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412b5350-2934-41d2-b82e-5fa4a13e80a9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.030133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412b5350-2934-41d2-b82e-5fa4a13e80a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.030416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412b5350-2934-41d2-b82e-5fa4a13e80a9 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.031244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412b5350-2934-41d2-b82e-5fa4a13e80a9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:24.031561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412b5350-2934-41d2-b82e-5fa4a13e80a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.032996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412b5350-2934-41d2-b82e-5fa4a13e80a9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:24.042678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 31/68] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:24.044029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41aaf21-a676-4189-9d61-1d478134020f None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.044322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41aaf21-a676-4189-9d61-1d478134020f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.044604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41aaf21-a676-4189-9d61-1d478134020f None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.046115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 32/69] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:24.047995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8016671-382c-464c-aece-d2f9df83b4fa None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.047995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8016671-382c-464c-aece-d2f9df83b4fa None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.047995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8016671-382c-464c-aece-d2f9df83b4fa None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.049031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 33/70] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:24.049849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8016671-382c-464c-aece-d2f9df83b4fa None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.050356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3161ad8-7557-4019-9bca-6868a233b6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.051405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3161ad8-7557-4019-9bca-6868a233b6ee None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.051897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3161ad8-7557-4019-9bca-6868a233b6ee None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.052483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3161ad8-7557-4019-9bca-6868a233b6ee None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.053477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3161ad8-7557-4019-9bca-6868a233b6ee None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:24.053988 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3161ad8-7557-4019-9bca-6868a233b6ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.055658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3161ad8-7557-4019-9bca-6868a233b6ee None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:24.064978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 38/71] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:24.066713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b529a5f-1569-4c2c-a8db-082bfa2cc6e4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.067009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b529a5f-1569-4c2c-a8db-082bfa2cc6e4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.067295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b529a5f-1569-4c2c-a8db-082bfa2cc6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.085137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-6b529a5f-1569-4c2c-a8db-082bfa2cc6e4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:24.086381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6b529a5f-1569-4c2c-a8db-082bfa2cc6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:24.095046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.136144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.166323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 39/72] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:24.167996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24fcfa7-89d4-4c36-98bd-300b18e52330 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.168119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8016671-382c-464c-aece-d2f9df83b4fa None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.168545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24fcfa7-89d4-4c36-98bd-300b18e52330 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.168980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24fcfa7-89d4-4c36-98bd-300b18e52330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.186224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e24fcfa7-89d4-4c36-98bd-300b18e52330 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:24.187447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e24fcfa7-89d4-4c36-98bd-300b18e52330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:24.215562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.216443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.216736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.217023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] PATH_INFO: `/v3/users/a7d8b4db44114e1ba544d78dc12ef30a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.218237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] RBAC: Authorizing `identity:get_user(user_id=a7d8b4db44114e1ba544d78dc12ef30a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.219749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:24.221394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 34/73] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:24 2022] GET /identity/v3/users/a7d8b4db44114e1ba544d78dc12ef30a => generated 247 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:24.222167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.261645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 40/74] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:24.263168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ba89e-7a38-4a42-a678-f58b36b3800a None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.263470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ba89e-7a38-4a42-a678-f58b36b3800a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.263757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ba89e-7a38-4a42-a678-f58b36b3800a None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.265183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 41/75] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:24.265968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ba89e-7a38-4a42-a678-f58b36b3800a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.343683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba113b3-1c7f-4a0b-a117-9915872b5a89 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.347757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.380099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.380983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.381268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.381543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.383488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ba89e-7a38-4a42-a678-f58b36b3800a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.391219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.397001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.398538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:24.400794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 35/76] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:24 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:24.401571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.419926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62dfbc-2e56-4cc8-a99b-81aab4eba410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.420737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62dfbc-2e56-4cc8-a99b-81aab4eba410 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.421022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62dfbc-2e56-4cc8-a99b-81aab4eba410 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.421333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62dfbc-2e56-4cc8-a99b-81aab4eba410 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.422553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62dfbc-2e56-4cc8-a99b-81aab4eba410 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.424020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62dfbc-2e56-4cc8-a99b-81aab4eba410 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:24.425842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 42/77] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:24.426834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6bdb61-92d1-4a0c-8e50-eba691c9c899 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.427129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6bdb61-92d1-4a0c-8e50-eba691c9c899 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.427402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6bdb61-92d1-4a0c-8e50-eba691c9c899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.444964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fe6bdb61-92d1-4a0c-8e50-eba691c9c899 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:24.446226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fe6bdb61-92d1-4a0c-8e50-eba691c9c899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:24.505603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a910668-aa26-4bbf-94f5-5ea8972b22d0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.517905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 43/78] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:24.518779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6bdb61-92d1-4a0c-8e50-eba691c9c899 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.542589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.543255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.543545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.543827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.576072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.616573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.618696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:24.634868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6bdb61-92d1-4a0c-8e50-eba691c9c899 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.637289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.673780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.674631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.674917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.675319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.685800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:24.694215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:24.695366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:24.696708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 44/79] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:25:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:24.697408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.722944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 36/80] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:24.723788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.789118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.805032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7877e-4bac-48fe-9c82-441d85c9b01c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.829873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0634ab-5303-499e-971e-567ee7231a77 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:24.839887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.840554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.840856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.841144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.866406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:24.867112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:24.867396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:24.867743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:24.868475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.869930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:24.872694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:24.897778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 37/81] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:25:24 2022] POST /identity/v3/users => generated 289 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:24.897778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:24.926485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:24.928136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.016246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e794ad4-66b2-425d-817e-8cb6aa2ee4fb None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.036176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:25.053034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 45/82] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:25.053831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.060769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.061589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.061887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.062183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.076243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:25.084191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:25.085092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:25.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 38/83] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:25.087129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.132248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:25.188365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e323c77d-4389-484e-bc97-788552298fcb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.232431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2bc7-8889-4217-8413-519d2aea74cd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.234995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.235684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.235973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.236253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.237018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:25.237327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:25.239649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.251069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 46/84] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:25.251939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.285552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.287485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.287991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.288711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.291502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:25.293597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.296424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 39/85] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:25.297451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.363055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:25.374877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98277109-0742-4371-ad44-baa376e3b57d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.413285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.414259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.415048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.415361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.416652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:25.418261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.426311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 47/86] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:25.426311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.426842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:25.450151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb3b2c7-88f7-4197-bfd7-2bcac23d2b6e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.495543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.496243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.496540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.496829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.497613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:25.497931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:25.499411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.509339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 40/87] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:25.510208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.548260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e925c82-983f-4210-9141-fe296a2aac1e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.589215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.590777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.591881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.592410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.594544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:25.597353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.599802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 48/88] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:25.601446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.629742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0957bde6-0181-4dcb-8b4d-33f8a426c9da None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.640319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:25.678252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.679243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.679740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.680784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.682415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=a7d8b4db44114e1ba544d78dc12ef30a) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:25.683242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:25.685616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.690980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:25.700288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 41/89] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/role_assignments?user.id=a7d8b4db44114e1ba544d78dc12ef30a&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:25.700288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.776892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7b67a-da68-452c-8ee9-0bd1fddf4dd2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.831412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.832101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.832390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.832672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:25.833594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=a6d8547cc7c84d159b1e737f172c73fd) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:25.833913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:25.835442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:25.858065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 49/90] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/role_assignments?user.id=a6d8547cc7c84d159b1e737f172c73fd&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=0d2588012d264232b288e3d89e4579c3 => generated 259 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:25.858805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:25.923007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e83c96-221e-471a-a0aa-40e4e5ccab83 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:25.997156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:25.998089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:25.998378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:25.998683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:26.001207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:26.002724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:26.004693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 42/91] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:26.011035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:26.071451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c4803-c799-4933-adad-8b7d3d0fdc0a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:26.099662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:26.126234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:26.145658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:26.146339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:26.146647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:26.146929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:26.220574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b3bd90-cceb-4468-a0dd-621ead045491 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:26.274010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:26.275527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:26.288216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:26.288908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:26.289195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:26.290080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:26.439693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:26.441208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:26.464153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 50/92] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 606 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:26.469892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:26.521665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:26.545347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:26.675110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 43/93] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 665 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 11 14:25:26.681809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:26.692491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d716e3-d5c9-4f15-995c-fcd1c8ca5ebb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:26.763437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:26.764337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:26.764630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:26.764915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:26.771252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:26.772943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:26.774970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 51/94] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:26 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:26.781093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:26.865443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53d5a8-758c-4a71-ae2a-fbbc49c19ab1 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:26.940253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:26.941143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:26.941434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:26.941715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] PATH_INFO: `/v3/projects/e5b09a71f6524ae6bec581e8037d5e51` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:26.942989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] RBAC: Authorizing `identity:get_project(project_id=e5b09a71f6524ae6bec581e8037d5e51)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:26.944482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:26.951040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 44/95] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:26 2022] GET /identity/v3/projects/e5b09a71f6524ae6bec581e8037d5e51 => generated 312 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:26.965774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:26.985093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:26.994967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:26.998311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2f9ef5-ca52-4808-96e9-7cd3e9debafd None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:27.107268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d1d46f-59a0-440c-8a67-ac4e29c4a00e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:27.108201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d1d46f-59a0-440c-8a67-ac4e29c4a00e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:27.108639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d1d46f-59a0-440c-8a67-ac4e29c4a00e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:27.109082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d1d46f-59a0-440c-8a67-ac4e29c4a00e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:27.110083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d1d46f-59a0-440c-8a67-ac4e29c4a00e None admin] RBAC: Adding query filter params (role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=a6d8547cc7c84d159b1e737f172c73fd) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:27.110570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d1d46f-59a0-440c-8a67-ac4e29c4a00e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:27.112213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d1d46f-59a0-440c-8a67-ac4e29c4a00e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:27.122822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 52/96] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:26 2022] GET /identity/v3/role_assignments?user.id=a6d8547cc7c84d159b1e737f172c73fd&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 259 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:27.167322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984d96e-9941-48a6-9ea3-aba4436a7b46 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:27.206376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f674c7b-5ef5-4f29-8282-e10a4ab4226b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:27.207084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f674c7b-5ef5-4f29-8282-e10a4ab4226b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:27.207368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f674c7b-5ef5-4f29-8282-e10a4ab4226b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:27.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f674c7b-5ef5-4f29-8282-e10a4ab4226b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:27.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f674c7b-5ef5-4f29-8282-e10a4ab4226b None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=e5b09a71f6524ae6bec581e8037d5e51, user.id=a6d8547cc7c84d159b1e737f172c73fd) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:27.208808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f674c7b-5ef5-4f29-8282-e10a4ab4226b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:27.210401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f674c7b-5ef5-4f29-8282-e10a4ab4226b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:27.222038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 45/97] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:26 2022] GET /identity/v3/role_assignments?user.id=a6d8547cc7c84d159b1e737f172c73fd&scope.project.id=e5b09a71f6524ae6bec581e8037d5e51&role.id=0d2588012d264232b288e3d89e4579c3 => generated 259 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:27.378545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f90072-c9ca-4ab7-900f-fe9106b52429 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:27.379085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f90072-c9ca-4ab7-900f-fe9106b52429 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:27.379516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f90072-c9ca-4ab7-900f-fe9106b52429 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:27.381204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 53/98] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:27.387352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e327df-3c17-4b7b-b4f7-9d2309fd3c06 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:27.387654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e327df-3c17-4b7b-b4f7-9d2309fd3c06 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:27.387944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e327df-3c17-4b7b-b4f7-9d2309fd3c06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:27.415680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d5e327df-3c17-4b7b-b4f7-9d2309fd3c06 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:27.416999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d5e327df-3c17-4b7b-b4f7-9d2309fd3c06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:27.509411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 46/99] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:27.552466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f90072-c9ca-4ab7-900f-fe9106b52429 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:27.697999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f90072-c9ca-4ab7-900f-fe9106b52429 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:27.738022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:27.738892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:27.739179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:27.739460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:27.754943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:27.762993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:27.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:27.764601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 54/100] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:27.771777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e327df-3c17-4b7b-b4f7-9d2309fd3c06 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:27.774626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:27.905824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e327df-3c17-4b7b-b4f7-9d2309fd3c06 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:27.949204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:27.950503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:27.950988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:27.951460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:27.952409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:27.952956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:27.954561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:27.970218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 47/101] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:27.976134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:28.011732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:28.104492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f747f0b-25cc-40e6-91f6-6b83a739bc3a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:28.146083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1ed937-1321-444a-baca-7c5531b848d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:28.147018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1ed937-1321-444a-baca-7c5531b848d8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.147312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1ed937-1321-444a-baca-7c5531b848d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.147597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1ed937-1321-444a-baca-7c5531b848d8 None admin] PATH_INFO: `/v3/users/f8fca419d65b408f8c418ceedaebec9e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.171268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1ed937-1321-444a-baca-7c5531b848d8 None admin] RBAC: Authorizing `identity:get_user(user_id=f8fca419d65b408f8c418ceedaebec9e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:28.172947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1ed937-1321-444a-baca-7c5531b848d8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:28.174847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 55/102] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:27 2022] GET /identity/v3/users/f8fca419d65b408f8c418ceedaebec9e => generated 289 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:28.183568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:28.234813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797a090-0c8b-4483-9768-6b98f27682d9 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.235460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797a090-0c8b-4483-9768-6b98f27682d9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.235765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797a090-0c8b-4483-9768-6b98f27682d9 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.237623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 56/103] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:28.244477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66ab7b-8326-4ec2-a05f-fcadbc90896d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.244915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66ab7b-8326-4ec2-a05f-fcadbc90896d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.245325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66ab7b-8326-4ec2-a05f-fcadbc90896d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.267715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-aa66ab7b-8326-4ec2-a05f-fcadbc90896d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:28.269180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-aa66ab7b-8326-4ec2-a05f-fcadbc90896d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:28.311875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:28.356171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 57/104] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:28.394698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9f8291-faef-4c58-a985-2d61d16de3a3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:28.422870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66ab7b-8326-4ec2-a05f-fcadbc90896d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:28.434807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:28.446733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7ea14c-14e0-4056-953e-048134321da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:28.447507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ea14c-14e0-4056-953e-048134321da5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.447798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ea14c-14e0-4056-953e-048134321da5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.448087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7ea14c-14e0-4056-953e-048134321da5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.555177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66ab7b-8326-4ec2-a05f-fcadbc90896d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:28.573882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7ea14c-14e0-4056-953e-048134321da5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:28.575804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7ea14c-14e0-4056-953e-048134321da5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:28.610812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:28.611762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.612057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.612338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.613931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:28.615518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:28.625384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 58/105] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:28.641384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:28.690365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:28.772170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:28.777295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 48/106] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 594 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:28.779480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9620696-521b-4aae-ad74-9d3a730b5ff6 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.782671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9620696-521b-4aae-ad74-9d3a730b5ff6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.783146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9620696-521b-4aae-ad74-9d3a730b5ff6 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.784764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 49/107] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:28 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:28.787719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9620696-521b-4aae-ad74-9d3a730b5ff6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:28.828387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c1c514-d19d-40db-b68c-f9af299ae0eb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:28.889141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c73e6b8-34e3-404c-a49f-94cedd3a8228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:28.890231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73e6b8-34e3-404c-a49f-94cedd3a8228 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.890675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73e6b8-34e3-404c-a49f-94cedd3a8228 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.891066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73e6b8-34e3-404c-a49f-94cedd3a8228 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.892318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c73e6b8-34e3-404c-a49f-94cedd3a8228 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:28.893917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c73e6b8-34e3-404c-a49f-94cedd3a8228 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:28.943991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 59/108] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:25:28 2022] POST /identity/v3/projects => generated 302 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:25:28.946502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bced5d1c-0633-44cb-99da-a3bfc51c28de None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:28.946980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bced5d1c-0633-44cb-99da-a3bfc51c28de None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:28.947397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bced5d1c-0633-44cb-99da-a3bfc51c28de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:28.965613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9620696-521b-4aae-ad74-9d3a730b5ff6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:28.971972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-bced5d1c-0633-44cb-99da-a3bfc51c28de None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:28.973693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bced5d1c-0633-44cb-99da-a3bfc51c28de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:29.037790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e84916-5a91-4c37-b2ec-966a1b8b7345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:29.038930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e84916-5a91-4c37-b2ec-966a1b8b7345 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.039321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e84916-5a91-4c37-b2ec-966a1b8b7345 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.040227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e84916-5a91-4c37-b2ec-966a1b8b7345 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.045460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e84916-5a91-4c37-b2ec-966a1b8b7345 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:29.047146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e84916-5a91-4c37-b2ec-966a1b8b7345 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:29.049589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 50/109] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:28 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:29.050889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13836c2-d784-4532-9c73-769cf054e02b None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.051256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13836c2-d784-4532-9c73-769cf054e02b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.051636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13836c2-d784-4532-9c73-769cf054e02b None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.053117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 51/110] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:29.059665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13836c2-d784-4532-9c73-769cf054e02b None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:29.062037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 60/111] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:29.063889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec655d-91b0-4634-863d-a369bfd1969f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.064381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec655d-91b0-4634-863d-a369bfd1969f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.064889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec655d-91b0-4634-863d-a369bfd1969f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.098822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-23ec655d-91b0-4634-863d-a369bfd1969f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:29.105706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-23ec655d-91b0-4634-863d-a369bfd1969f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:29.123193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:29.224866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:29.247604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13836c2-d784-4532-9c73-769cf054e02b None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:29.261902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 61/112] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:29.263390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec655d-91b0-4634-863d-a369bfd1969f None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:29.300477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:29.301156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.301446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.301802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.303734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] RBAC: Adding query filter params (role.id=43637b70b75a4d028f9ca3ea333735b5, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=f8fca419d65b408f8c418ceedaebec9e) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:29.307057 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:29.308563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:29.318554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 52/113] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:29 2022] GET /identity/v3/role_assignments?user.id=f8fca419d65b408f8c418ceedaebec9e&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=43637b70b75a4d028f9ca3ea333735b5 => generated 259 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:29.320256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:29.413211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec655d-91b0-4634-863d-a369bfd1969f None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:29.434131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:29.442447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24477f56-c472-4200-a11a-8dd2c73c1977 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:29.463461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:29.464649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.465091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.465637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.480502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:29.483219 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:29.484165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.484559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.484943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:29.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:29.495519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:29.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:29.502464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:29.503149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 62/114] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:29.508989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:29.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:29.510411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:29.511316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 53/115] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:29.517438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:29.596100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:29.670132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1acab9c-7ffc-4165-a5f1-9726eb48029b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:29.729338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7526437-627f-4907-a7f6-da3c7e7072fe None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:29.764206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:29.764917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.765216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.765507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.766290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:29.766635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:29.768061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:29.785808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 54/116] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:29.796168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:29.801758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:29.802491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.802789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.803071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.803847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:29.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:29.805675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:29.820215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 63/117] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:29.826890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:29.830946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:29.958563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f729d7-0201-4cb8-9dc2-0a09c64deaad None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:29.996228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:29.997130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:29.997424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:29.997713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:29.998539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca8b5ef-68c2-4a82-8b8f-a59b7cbb4b4f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:29.998999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:30.000500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:30.002197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 55/118] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:29 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:30.008876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:30.018839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:30.068417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:30.069407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.069701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.069982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] PATH_INFO: `/v3/users/a7d8b4db44114e1ba544d78dc12ef30a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.073094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] RBAC: Authorizing `identity:get_user(user_id=a7d8b4db44114e1ba544d78dc12ef30a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:30.074946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:30.080431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 64/119] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:29 2022] GET /identity/v3/users/a7d8b4db44114e1ba544d78dc12ef30a => generated 247 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:30.082470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:30.166829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:30.188079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18032a30-f558-4544-ac23-ebc6b39c9a2d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:30.235470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:30.236331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.236623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.236907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.238474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:30.239970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:30.241960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 56/120] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:30 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:30.248097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:30.348754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d62f9c-d121-4a38-900a-e739d0a27c8a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:30.359824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0f683f-322b-452c-84e0-692691a3271d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:30.387794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a7be8f-1c64-481d-8333-53136ac1a2bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:30.388679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a7be8f-1c64-481d-8333-53136ac1a2bc None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.388968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a7be8f-1c64-481d-8333-53136ac1a2bc None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.389378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a7be8f-1c64-481d-8333-53136ac1a2bc None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.390954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a7be8f-1c64-481d-8333-53136ac1a2bc None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:30.392546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a7be8f-1c64-481d-8333-53136ac1a2bc None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:30.394486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 65/121] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:30 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:30.395999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c514b3-6504-4334-8064-4f8e95e48713 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.396296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c514b3-6504-4334-8064-4f8e95e48713 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.396583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c514b3-6504-4334-8064-4f8e95e48713 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.397883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 66/122] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:30.399406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4c3bbf-f9ea-412b-a618-37812c46eaaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:30.400887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4c3bbf-f9ea-412b-a618-37812c46eaaa None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.401367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4c3bbf-f9ea-412b-a618-37812c46eaaa None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.401835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4c3bbf-f9ea-412b-a618-37812c46eaaa None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/a6d8547cc7c84d159b1e737f172c73fd/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.402542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c514b3-6504-4334-8064-4f8e95e48713 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:30.404455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4c3bbf-f9ea-412b-a618-37812c46eaaa None admin] RBAC: Authorizing `identity:create_grant(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a, user_id=a6d8547cc7c84d159b1e737f172c73fd, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:30.406651 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4c3bbf-f9ea-412b-a618-37812c46eaaa None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:30.437062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 57/123] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:30 2022] PUT /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/a6d8547cc7c84d159b1e737f172c73fd/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:25:30.439090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9986d368-d22d-4fa7-b1fa-901d0ad4857f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.439585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9986d368-d22d-4fa7-b1fa-901d0ad4857f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.440045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9986d368-d22d-4fa7-b1fa-901d0ad4857f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.471839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9986d368-d22d-4fa7-b1fa-901d0ad4857f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:30.473680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9986d368-d22d-4fa7-b1fa-901d0ad4857f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:30.486657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:30.490788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:30.586389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c514b3-6504-4334-8064-4f8e95e48713 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:30.597358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 58/124] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:30.630511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9986d368-d22d-4fa7-b1fa-901d0ad4857f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:30.631077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be31902-cf88-4772-a440-d21a62dd09ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:30.631994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be31902-cf88-4772-a440-d21a62dd09ee None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.632281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be31902-cf88-4772-a440-d21a62dd09ee None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.632558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be31902-cf88-4772-a440-d21a62dd09ee None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/a7d8b4db44114e1ba544d78dc12ef30a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.635191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be31902-cf88-4772-a440-d21a62dd09ee None admin] RBAC: Authorizing `identity:create_grant(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a, user_id=a7d8b4db44114e1ba544d78dc12ef30a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:30.636675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be31902-cf88-4772-a440-d21a62dd09ee None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:30.682421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 67/125] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:30 2022] PUT /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/a7d8b4db44114e1ba544d78dc12ef30a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:25:30.749273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb05c05-4269-4b76-891b-2faf2788e9fc None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.752238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb05c05-4269-4b76-891b-2faf2788e9fc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.752753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb05c05-4269-4b76-891b-2faf2788e9fc None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.755276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 68/126] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:30 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:30.764295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999d19f3-56a8-4b2c-af2f-e05f4f6d58d7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.764609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999d19f3-56a8-4b2c-af2f-e05f4f6d58d7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.764876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999d19f3-56a8-4b2c-af2f-e05f4f6d58d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.790929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-999d19f3-56a8-4b2c-af2f-e05f4f6d58d7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:30.792183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-999d19f3-56a8-4b2c-af2f-e05f4f6d58d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:30.826902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9986d368-d22d-4fa7-b1fa-901d0ad4857f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:30.868081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:30.869170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:30.869359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:30.869644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:30.874525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 69/127] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:30.884001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:30.892722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:30.893731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:30.894595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 59/128] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:25:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:30.901093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999d19f3-56a8-4b2c-af2f-e05f4f6d58d7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:30.919985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:30.925930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:30.960598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:31.049168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d82e41-e44f-4018-b3c7-7ca1a2bbb997 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:31.087356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:31.088425 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:31.088925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:31.089422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:31.090404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:31.090957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:31.092585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:31.102154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 60/129] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:25:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:31.110338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:31.136496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999d19f3-56a8-4b2c-af2f-e05f4f6d58d7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:31.202537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:31.216381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:31.217218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:31.217511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:31.217794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:31.239968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13201b3b-b1f5-46ba-ba08-c5bf1bcd726d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:31.245891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:31.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:31.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:31.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:31.259813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 70/130] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 359 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:31.267502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:31.302937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:31.303808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:31.304095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:31.304496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:31.307583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:31.309237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:31.311201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 61/131] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:31 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:31.323413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:31.468819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:31.474311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff0d34-eacc-45f9-934c-d43e4f36b909 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:31.495610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee6f49-1466-4eb1-bf9c-61a1e2bdb787 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:31.551612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:31.552314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:31.552605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:31.552887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:31.553712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:31.553968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:31.555521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:31.555787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:31.556689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:31.557105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:31.557399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:31.563294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:31.564830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:31.566714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 62/132] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:31 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:31.573626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:31.574995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 71/133] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:31.591317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:31.592511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:31.690191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9340137f-f55b-49fa-9580-fce6cfe3e447 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:31.739237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:31.740174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:31.740478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:31.740753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/a6d8547cc7c84d159b1e737f172c73fd/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:31.749619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] RBAC: Authorizing `identity:create_grant(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a, user_id=a6d8547cc7c84d159b1e737f172c73fd, role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:31.751155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:31.776510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 63/134] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:31 2022] PUT /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/a6d8547cc7c84d159b1e737f172c73fd/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:25:31.793458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e94ac-1b78-41ab-98f3-7cb8ecea3c60 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:31.847292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32b2575-4364-46a5-8e24-68a49ea8693f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:31.848114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32b2575-4364-46a5-8e24-68a49ea8693f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:31.848402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32b2575-4364-46a5-8e24-68a49ea8693f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:31.848681 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32b2575-4364-46a5-8e24-68a49ea8693f None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:31.850030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32b2575-4364-46a5-8e24-68a49ea8693f None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:31.851545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32b2575-4364-46a5-8e24-68a49ea8693f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:31.853290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 72/135] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:31 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:31.858922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:31.986741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:32.032701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f6b40-b129-4a89-b8a5-13bb69ff1b1f None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.033038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f6b40-b129-4a89-b8a5-13bb69ff1b1f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.033282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f6b40-b129-4a89-b8a5-13bb69ff1b1f None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.034992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 73/136] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:32.042182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b530231-abff-45d0-aedb-8877a7fde96c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:32.042733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fc878f-d211-4605-aee2-34c6d43f7291 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.043093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fc878f-d211-4605-aee2-34c6d43f7291 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.043386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fc878f-d211-4605-aee2-34c6d43f7291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.082002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-27fc878f-d211-4605-aee2-34c6d43f7291 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:32.083421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-27fc878f-d211-4605-aee2-34c6d43f7291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:32.091761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:32.092675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.092951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.093240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] PATH_INFO: `/v3/projects/e5b09a71f6524ae6bec581e8037d5e51` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.094559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] RBAC: Authorizing `identity:get_project(project_id=e5b09a71f6524ae6bec581e8037d5e51)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:32.100998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:32.103370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 64/137] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:31 2022] GET /identity/v3/projects/e5b09a71f6524ae6bec581e8037d5e51 => generated 312 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:32.111532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:32.129916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:32.206940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 74/138] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:32.249599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fc878f-d211-4605-aee2-34c6d43f7291 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:32.253869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fbf4eb-70f1-4507-9a49-fddf1cc32266 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:32.307741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:32.325939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:32.326908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.327473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.327760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] PATH_INFO: `/v3/projects/e5b09a71f6524ae6bec581e8037d5e51/users/a6d8547cc7c84d159b1e737f172c73fd/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.332146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5b09a71f6524ae6bec581e8037d5e51, user_id=a6d8547cc7c84d159b1e737f172c73fd, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:32.333797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:32.357383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 65/139] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:32 2022] PUT /identity/v3/projects/e5b09a71f6524ae6bec581e8037d5e51/users/a6d8547cc7c84d159b1e737f172c73fd/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:25:32.409804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fc878f-d211-4605-aee2-34c6d43f7291 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:32.450647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f1787-d18b-4c04-aa9f-358d4b26e908 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:32.451468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f1787-d18b-4c04-aa9f-358d4b26e908 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.451756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f1787-d18b-4c04-aa9f-358d4b26e908 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.452037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f1787-d18b-4c04-aa9f-358d4b26e908 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.453319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f1787-d18b-4c04-aa9f-358d4b26e908 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:32.454890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f1787-d18b-4c04-aa9f-358d4b26e908 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:32.456696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 75/140] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:32.463313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:32.605748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fafc51-f05e-4c80-961f-783e35a6f311 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:32.644004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964204e0-2acb-4dcd-a3dc-6f284c1150e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:32.644929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964204e0-2acb-4dcd-a3dc-6f284c1150e0 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.645366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964204e0-2acb-4dcd-a3dc-6f284c1150e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.645824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964204e0-2acb-4dcd-a3dc-6f284c1150e0 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.646753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964204e0-2acb-4dcd-a3dc-6f284c1150e0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:32.648416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964204e0-2acb-4dcd-a3dc-6f284c1150e0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:32.674243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 66/141] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:25:32 2022] POST /identity/v3/users => generated 283 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:32.783142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f2996-ca4a-4fb3-8f43-7f78a624eaf2 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.783600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f2996-ca4a-4fb3-8f43-7f78a624eaf2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.784004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f2996-ca4a-4fb3-8f43-7f78a624eaf2 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.785744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 76/142] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:32.801058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3e90f2-6cbb-46e1-af54-7156a4962d5e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:32.801704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3e90f2-6cbb-46e1-af54-7156a4962d5e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:32.802247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3e90f2-6cbb-46e1-af54-7156a4962d5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:32.820420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9f3e90f2-6cbb-46e1-af54-7156a4962d5e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:32.826661 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9f3e90f2-6cbb-46e1-af54-7156a4962d5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:32.875809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:32.912620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 67/143] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:32.938904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f2996-ca4a-4fb3-8f43-7f78a624eaf2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:32.972770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:33.072390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f2996-ca4a-4fb3-8f43-7f78a624eaf2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:33.118754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:33.119732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:33.120023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:33.120303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:33.135510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:33.144432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:33.146182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 77/144] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:33.151540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3e90f2-6cbb-46e1-af54-7156a4962d5e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:33.309983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3e90f2-6cbb-46e1-af54-7156a4962d5e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:33.365832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:33.365832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:33.365832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:33.365832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:33.366229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] RBAC: Adding query filter params (name=reader) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:33.366532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:33.368546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:33.390163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 68/145] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:33.395547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:33.538261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:33.559143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8870b8e0-a892-4111-bb3b-5c8aa969c2b9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:33.607375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ec7f7d-74e0-44eb-816f-fd70e42d27be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:33.608193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ec7f7d-74e0-44eb-816f-fd70e42d27be None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:33.608479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ec7f7d-74e0-44eb-816f-fd70e42d27be None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:33.608759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ec7f7d-74e0-44eb-816f-fd70e42d27be None admin] PATH_INFO: `/v3/users/f8fca419d65b408f8c418ceedaebec9e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:33.610042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ec7f7d-74e0-44eb-816f-fd70e42d27be None admin] RBAC: Authorizing `identity:get_user(user_id=f8fca419d65b408f8c418ceedaebec9e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:33.611559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ec7f7d-74e0-44eb-816f-fd70e42d27be None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:33.613467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 78/146] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:33 2022] GET /identity/v3/users/f8fca419d65b408f8c418ceedaebec9e => generated 289 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:33.618836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:33.682056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:33.763260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa46826-68b3-4a5b-ae83-309da963ab1b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:33.763568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa46826-68b3-4a5b-ae83-309da963ab1b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:33.763854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa46826-68b3-4a5b-ae83-309da963ab1b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:33.765541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 79/147] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:33.767818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7811f2-a47a-4395-b294-2007b406dc5f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:33.770971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b481b-13d8-4ee9-a245-d65a370a9703 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:33.771259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b481b-13d8-4ee9-a245-d65a370a9703 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:33.771538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b481b-13d8-4ee9-a245-d65a370a9703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:33.811631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-940b481b-13d8-4ee9-a245-d65a370a9703 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:33.812859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-940b481b-13d8-4ee9-a245-d65a370a9703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:33.835647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:33.835647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:33.835647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:33.836498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:33.838035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:33.840000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:33.842090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 69/148] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:33 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:33.847773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:33.997955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f02da4-de92-4efe-ac4e-2a8176109cb6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:34.010800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:34.044339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 80/149] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:34.060343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24b0b52-ef54-4ea6-a9db-4986414e32de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:34.061356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24b0b52-ef54-4ea6-a9db-4986414e32de None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:34.061735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24b0b52-ef54-4ea6-a9db-4986414e32de None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:34.062090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24b0b52-ef54-4ea6-a9db-4986414e32de None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/f8fca419d65b408f8c418ceedaebec9e/roles/43637b70b75a4d028f9ca3ea333735b5` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:34.064250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24b0b52-ef54-4ea6-a9db-4986414e32de None admin] RBAC: Authorizing `identity:create_grant(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a, user_id=f8fca419d65b408f8c418ceedaebec9e, role_id=43637b70b75a4d028f9ca3ea333735b5)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:34.065831 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24b0b52-ef54-4ea6-a9db-4986414e32de None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:34.085357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 70/150] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:33 2022] PUT /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/users/f8fca419d65b408f8c418ceedaebec9e/roles/43637b70b75a4d028f9ca3ea333735b5 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:25:34.094087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940b481b-13d8-4ee9-a245-d65a370a9703 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:34.177726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eba6b2c-09fd-4110-b962-6850c982e00a None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:34.177988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eba6b2c-09fd-4110-b962-6850c982e00a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:34.178565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eba6b2c-09fd-4110-b962-6850c982e00a None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:34.184879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 71/151] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:34 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:34.191210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b6bf4-0a14-4930-bb6a-3486fbb6c0e5 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:34.191510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b6bf4-0a14-4930-bb6a-3486fbb6c0e5 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:34.191780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b6bf4-0a14-4930-bb6a-3486fbb6c0e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:34.205718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:34.220043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8d7b6bf4-0a14-4930-bb6a-3486fbb6c0e5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:34.221397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8d7b6bf4-0a14-4930-bb6a-3486fbb6c0e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:34.246828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940b481b-13d8-4ee9-a245-d65a370a9703 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:34.333844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 72/152] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:34.357101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:34.357930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:34.358211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:34.358510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:34.379639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:34.388155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:34.389060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:34.389818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 81/153] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:34.397690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b6bf4-0a14-4930-bb6a-3486fbb6c0e5 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:34.399293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:34.538891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:34.544994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b6bf4-0a14-4930-bb6a-3486fbb6c0e5 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:34.600190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:34.600807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:34.601104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:34.601390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:34.602728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:34.603050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:34.604711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:34.622788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 73/154] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:34.624423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:34.625985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ad8fbf-3e94-4deb-a4b7-3f381ec9b5ed None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:34.684504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:34.685392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:34.685686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:34.685967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:34.696308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:34.708893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:34.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:34.724469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:34.725441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 82/155] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:34.735501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:34.781053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee84d5d-9806-417b-986e-ef15d47f0c11 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:34.837154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:34.838064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:34.838399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:34.838814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] PATH_INFO: `/v3/users/a7d8b4db44114e1ba544d78dc12ef30a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:34.840209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] RBAC: Authorizing `identity:get_user(user_id=a7d8b4db44114e1ba544d78dc12ef30a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:34.841723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:34.847862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 74/156] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:34 2022] GET /identity/v3/users/a7d8b4db44114e1ba544d78dc12ef30a => generated 247 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:34.848781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:34.914932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:34.920202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949dc57e-9327-4008-a0d5-2edd992bb364 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:35.018397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:35.019177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:35.019464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:35.019742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:35.020572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:35.020883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:35.022382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:35.036395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 83/157] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:35.042764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:35.104521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6d04b7-1167-4c1d-bdab-defc39dafe65 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:35.128826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:35.171126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223e5813-b106-4782-8fef-d23ab013635f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:35.233029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:35.234314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:35.234829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:35.235313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:35.239251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:35.241053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:35.243585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 84/158] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:35 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:35.246453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:35.246453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:35.246453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:35.246453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:35.255662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:35.435628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:35.437075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:35.474746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ba641d-4b10-4edc-8ef4-bd5bcbbc7417 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:35.497895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:35.538990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:35.549074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dfac3a-b9ad-47c2-b3b1-0e198aa225ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:35.550317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dfac3a-b9ad-47c2-b3b1-0e198aa225ff None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:35.550834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dfac3a-b9ad-47c2-b3b1-0e198aa225ff None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:35.551330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dfac3a-b9ad-47c2-b3b1-0e198aa225ff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:35.701088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dfac3a-b9ad-47c2-b3b1-0e198aa225ff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:35.702012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 75/159] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 853 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 11 14:25:35.703455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dfac3a-b9ad-47c2-b3b1-0e198aa225ff None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:35.711927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:35.898519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e138e-edf1-45c6-b94e-cb8c2c90e5a7 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:35.914695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 85/160] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 658 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 11 14:25:35.919010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3b8ed2-2f95-4114-b6d5-094eefe94f88 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:35.919324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3b8ed2-2f95-4114-b6d5-094eefe94f88 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:35.919613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3b8ed2-2f95-4114-b6d5-094eefe94f88 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:35.921070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 86/161] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:35 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:35.921901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3b8ed2-2f95-4114-b6d5-094eefe94f88 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:35.954876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:35.970768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a860f22-16bf-4fd2-8521-cb372dea9afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:35.971657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a860f22-16bf-4fd2-8521-cb372dea9afc None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:35.971945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a860f22-16bf-4fd2-8521-cb372dea9afc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:35.972224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a860f22-16bf-4fd2-8521-cb372dea9afc None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:35.974225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a860f22-16bf-4fd2-8521-cb372dea9afc None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:35.977420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a860f22-16bf-4fd2-8521-cb372dea9afc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:35.979418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 76/162] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:35 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:35.980789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13af74d2-93a0-49c5-a744-6d03ccfe4d0f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:35.981085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13af74d2-93a0-49c5-a744-6d03ccfe4d0f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:35.981476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13af74d2-93a0-49c5-a744-6d03ccfe4d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:35.990891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:36.008973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-13af74d2-93a0-49c5-a744-6d03ccfe4d0f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:36.010207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-13af74d2-93a0-49c5-a744-6d03ccfe4d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:36.167178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3b8ed2-2f95-4114-b6d5-094eefe94f88 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:36.185227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 77/163] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:36.186787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13af74d2-93a0-49c5-a744-6d03ccfe4d0f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:36.229425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:36.230231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:36.230540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:36.230821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:36.232165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:36.233651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:36.235489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 87/164] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:35 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:36.236419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:36.434375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13af74d2-93a0-49c5-a744-6d03ccfe4d0f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:36.464282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:36.471619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565e13b4-36d5-4dac-87e5-9d1e4ad95e51 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:36.490811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:36.506954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:36.507631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:36.508447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:36.508735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:36.509593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=a7d8b4db44114e1ba544d78dc12ef30a) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:36.509914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:36.511662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:36.525206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 78/165] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:36 2022] GET /identity/v3/role_assignments?user.id=a7d8b4db44114e1ba544d78dc12ef30a&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:36.526892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:36.545336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1665c1f-118a-4e39-b4f6-a47682dbce14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:36.546227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1665c1f-118a-4e39-b4f6-a47682dbce14 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:36.546542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1665c1f-118a-4e39-b4f6-a47682dbce14 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:36.546877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1665c1f-118a-4e39-b4f6-a47682dbce14 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:36.569562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1665c1f-118a-4e39-b4f6-a47682dbce14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-d1665c1f-118a-4e39-b4f6-a47682dbce14 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:36.585851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:36.586891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:36.587933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 88/166] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:25:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:36.591792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f5b337-1c45-4ec5-916d-3d4eed589e00 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:36.592084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f5b337-1c45-4ec5-916d-3d4eed589e00 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:36.592370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f5b337-1c45-4ec5-916d-3d4eed589e00 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:36.596474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 89/167] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:36.597315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f5b337-1c45-4ec5-916d-3d4eed589e00 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:36.798527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde54c06-5667-48e0-9bd6-bf9d3fbcdb59 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:36.868178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f5b337-1c45-4ec5-916d-3d4eed589e00 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:36.889535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac05cbc-45d5-4104-b445-13867c73b8fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:36.890269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac05cbc-45d5-4104-b445-13867c73b8fb None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:36.890583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac05cbc-45d5-4104-b445-13867c73b8fb None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:36.890866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac05cbc-45d5-4104-b445-13867c73b8fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:36.891862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac05cbc-45d5-4104-b445-13867c73b8fb None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=a6d8547cc7c84d159b1e737f172c73fd) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:36.892181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac05cbc-45d5-4104-b445-13867c73b8fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:36.893649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac05cbc-45d5-4104-b445-13867c73b8fb None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:36.910968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 79/168] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:36 2022] GET /identity/v3/role_assignments?user.id=a6d8547cc7c84d159b1e737f172c73fd&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=0d2588012d264232b288e3d89e4579c3 => generated 608 bytes in 385 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:36.913102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b4cf3-2834-4844-a90b-2ac3101e27bd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:36.913397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b4cf3-2834-4844-a90b-2ac3101e27bd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:36.913687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b4cf3-2834-4844-a90b-2ac3101e27bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:36.951538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fd5b4cf3-2834-4844-a90b-2ac3101e27bd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:36.953120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fd5b4cf3-2834-4844-a90b-2ac3101e27bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:36.953953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:36.954735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:36.955026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:36.955304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:36.956568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:36.957073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:36.958710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:36.978595 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 90/169] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:25:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 381 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:36.984640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:37.020370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:37.052544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:37.089335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 80/170] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:37.167903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0852b08-1e9e-4006-a2c0-ef84272554e4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:37.183043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5b4cf3-2834-4844-a90b-2ac3101e27bd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:37.234109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:37.235038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.235319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.235594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:37.240947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:37.242507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:37.244502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 91/171] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:36 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:37.250182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:37.376746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5b4cf3-2834-4844-a90b-2ac3101e27bd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:37.382650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:37.396170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39cff2-3fb9-4f63-8bb2-df8d2c044fb9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:37.423291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:37.424114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.424393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.424768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:37.434258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:37.434952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.435234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.435511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:37.444285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:37.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:37.463314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:37.464778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 81/172] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:37.472445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:37.512108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:37.513623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:37.543716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:37.612989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:37.627008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 92/173] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:37.632316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:37.644844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12dbeb0-ed05-49b8-85cd-d0a131f8d894 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:37.714914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:37.715612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.715894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.716171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:37.717038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:37.717351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:37.718924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:37.735587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 82/174] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:37.741476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:37.779718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c40e0-d990-4a60-a3de-d016ad0e7a59 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:37.866592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c750eef-37bf-4755-8c56-26e133825aeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:37.868003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c750eef-37bf-4755-8c56-26e133825aeb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.868286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c750eef-37bf-4755-8c56-26e133825aeb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.868558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c750eef-37bf-4755-8c56-26e133825aeb None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:37.869781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c750eef-37bf-4755-8c56-26e133825aeb None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:37.871299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c750eef-37bf-4755-8c56-26e133825aeb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:37.873130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 93/175] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:37 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:37.874406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec43e48-8f90-4aee-957d-7c015bb7a200 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.874715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec43e48-8f90-4aee-957d-7c015bb7a200 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.874994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec43e48-8f90-4aee-957d-7c015bb7a200 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:37.876265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 94/176] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:37.881013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec43e48-8f90-4aee-957d-7c015bb7a200 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:37.929414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8400865-0b53-40ae-bb4d-0a5df7ef54db None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:37.932615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:37.943781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:37.979366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292a9ea-24d0-4dbf-9de1-072dc179a0f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:37.981890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292a9ea-24d0-4dbf-9de1-072dc179a0f9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.982624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292a9ea-24d0-4dbf-9de1-072dc179a0f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.983347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292a9ea-24d0-4dbf-9de1-072dc179a0f9 None admin] PATH_INFO: `/v3/users/a6d8547cc7c84d159b1e737f172c73fd` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:37.985564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292a9ea-24d0-4dbf-9de1-072dc179a0f9 None admin] RBAC: Authorizing `identity:get_user(user_id=a6d8547cc7c84d159b1e737f172c73fd)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:37.987862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292a9ea-24d0-4dbf-9de1-072dc179a0f9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:37.991430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 83/177] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:37 2022] GET /identity/v3/users/a6d8547cc7c84d159b1e737f172c73fd => generated 275 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:37.997204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48560b7-e067-4683-86dd-315f6cc48e42 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:37.997204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48560b7-e067-4683-86dd-315f6cc48e42 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:37.997204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48560b7-e067-4683-86dd-315f6cc48e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec43e48-8f90-4aee-957d-7c015bb7a200 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.024973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b48560b7-e067-4683-86dd-315f6cc48e42 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:38.026790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b48560b7-e067-4683-86dd-315f6cc48e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:38.046748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e43c53-0aa9-4bef-9f08-6edfc0baa13d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.047801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e43c53-0aa9-4bef-9f08-6edfc0baa13d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.048277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e43c53-0aa9-4bef-9f08-6edfc0baa13d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.048621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e43c53-0aa9-4bef-9f08-6edfc0baa13d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.049553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e43c53-0aa9-4bef-9f08-6edfc0baa13d None admin] RBAC: Adding query filter params (role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=a6d8547cc7c84d159b1e737f172c73fd) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:38.049959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e43c53-0aa9-4bef-9f08-6edfc0baa13d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:38.051734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e43c53-0aa9-4bef-9f08-6edfc0baa13d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:38.058960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 95/178] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:37 2022] GET /identity/v3/role_assignments?user.id=a6d8547cc7c84d159b1e737f172c73fd&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:38.060515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c98e6f-a5cb-484f-afd5-9487ad15885d None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.060903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c98e6f-a5cb-484f-afd5-9487ad15885d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.061281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c98e6f-a5cb-484f-afd5-9487ad15885d None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.062996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 96/179] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:38.067412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c98e6f-a5cb-484f-afd5-9487ad15885d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.142245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 84/180] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:38.146533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc910b7-0bbd-417d-9b35-a989c7d1d350 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.146533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc910b7-0bbd-417d-9b35-a989c7d1d350 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.146533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc910b7-0bbd-417d-9b35-a989c7d1d350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.170506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cbc910b7-0bbd-417d-9b35-a989c7d1d350 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:38.170506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cbc910b7-0bbd-417d-9b35-a989c7d1d350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:38.242761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c98e6f-a5cb-484f-afd5-9487ad15885d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.257413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 85/181] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:38.259283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c449b9df-c83b-4fd0-8fe8-569b83f19a8c None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.259563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c449b9df-c83b-4fd0-8fe8-569b83f19a8c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.260124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c449b9df-c83b-4fd0-8fe8-569b83f19a8c None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.261151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 86/182] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:38.261859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c449b9df-c83b-4fd0-8fe8-569b83f19a8c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.266037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:38.285807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.287361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.287361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.287361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.300432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:38.372414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:38.373998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:38.377663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c449b9df-c83b-4fd0-8fe8-569b83f19a8c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.417499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7438dda9-04e4-4b78-83d3-340a24911145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.418202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7438dda9-04e4-4b78-83d3-340a24911145 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.418520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7438dda9-04e4-4b78-83d3-340a24911145 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.418799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7438dda9-04e4-4b78-83d3-340a24911145 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.429202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7438dda9-04e4-4b78-83d3-340a24911145 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-7438dda9-04e4-4b78-83d3-340a24911145 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:38.437123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:38.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 87/183] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:38.444204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aab8f3-89bd-4752-b614-9f18561ea814 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.444953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aab8f3-89bd-4752-b614-9f18561ea814 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.444953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aab8f3-89bd-4752-b614-9f18561ea814 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.444953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 88/184] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:38.444953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e68d73c-0a1f-4ffe-a2d9-1ce998ed9f20 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.444953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e68d73c-0a1f-4ffe-a2d9-1ce998ed9f20 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.444953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e68d73c-0a1f-4ffe-a2d9-1ce998ed9f20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.462301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7e68d73c-0a1f-4ffe-a2d9-1ce998ed9f20 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:38.463527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7e68d73c-0a1f-4ffe-a2d9-1ce998ed9f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:38.491859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 97/185] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:38.496860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.513994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:38.531033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 89/186] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:38.533039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e68d73c-0a1f-4ffe-a2d9-1ce998ed9f20 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.557613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:38.594738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49233f8-51dd-4e77-afd7-0f92f7119133 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.629097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b5f570-6c8a-43a7-8dd1-fd6df8e3fbff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.630065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b5f570-6c8a-43a7-8dd1-fd6df8e3fbff None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.630353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b5f570-6c8a-43a7-8dd1-fd6df8e3fbff None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.630670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b5f570-6c8a-43a7-8dd1-fd6df8e3fbff None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.631849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b5f570-6c8a-43a7-8dd1-fd6df8e3fbff None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:38.634017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b5f570-6c8a-43a7-8dd1-fd6df8e3fbff None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:38.634119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e68d73c-0a1f-4ffe-a2d9-1ce998ed9f20 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.635683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 98/187] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:38.639687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009bf68-8128-425d-a161-c8e4d054333d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.639977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009bf68-8128-425d-a161-c8e4d054333d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.640323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009bf68-8128-425d-a161-c8e4d054333d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.658550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f009bf68-8128-425d-a161-c8e4d054333d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:38.659877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f009bf68-8128-425d-a161-c8e4d054333d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:38.667929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.668547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.668829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.669104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.669859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:38.670170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:38.671604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:38.680900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 90/188] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:38.681705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.730876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 99/189] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:38.731642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009bf68-8128-425d-a161-c8e4d054333d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.751112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:38.774542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:38.791297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1179a-0323-457e-87b2-5552ff8ebe34 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.826412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.827304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.827589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.827867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] PATH_INFO: `/v3/projects/e5b09a71f6524ae6bec581e8037d5e51` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.829068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] RBAC: Authorizing `identity:get_project(project_id=e5b09a71f6524ae6bec581e8037d5e51)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:38.830886 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:38.832284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 91/190] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:38 2022] GET /identity/v3/projects/e5b09a71f6524ae6bec581e8037d5e51 => generated 312 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:38.832284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.837751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009bf68-8128-425d-a161-c8e4d054333d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.870075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.870934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.871221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.871501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.881979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:38.890352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:38.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:38.892053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 100/191] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:25:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:38.892707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:38.938575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3913651d-c932-4c8e-b515-e53dd4674fa3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:38.972473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:38.975338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:38.976015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:38.976297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:38.977092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:38.978571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:38.980536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:38.997354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e053d7-b0e1-4f09-b759-192fb41a5674 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.003890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 92/192] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:25:38 2022] POST /identity/v3/users => generated 297 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:39.004737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.028369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:39.036028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.037076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.037361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.037640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] PATH_INFO: `/v3/users/1379dfebc09041b888bbf8a77ce59938` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.054513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] RBAC: Authorizing `identity:get_user(user_id=1379dfebc09041b888bbf8a77ce59938)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.056614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.062961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 101/193] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:38 2022] GET /identity/v3/users/1379dfebc09041b888bbf8a77ce59938 => generated 283 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:39.062961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.147648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95af1f14-6c8a-426b-97e4-a1bfcc8351db None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.179661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89553cce-207d-41ab-99a2-0cea6b4391c5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.196677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.197423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.197711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.197991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.210575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:39.219445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:39.220558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:39.221367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:39.223558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 93/194] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/roles/reader => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:39.224086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.224406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.224834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.225114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.225398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.226276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=e5b09a71f6524ae6bec581e8037d5e51, user.id=a6d8547cc7c84d159b1e737f172c73fd) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:39.226684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.228172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.232688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:39.242774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 102/195] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/role_assignments?user.id=a6d8547cc7c84d159b1e737f172c73fd&scope.project.id=e5b09a71f6524ae6bec581e8037d5e51&role.id=0d2588012d264232b288e3d89e4579c3 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:39.242774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.336310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:39.387268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95afb1f-0e71-4273-a236-20589a4cc89f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.426063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdaf45f-0dfb-4101-b8fe-80b6669a58b2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.438337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.442459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.442459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.442459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.442459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:39.442459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.442459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.452566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 94/196] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:39.453351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.477894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.478710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.479007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.479451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.561659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:39.564707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.566551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.586897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afe8171-0301-41c0-ad23-7dada319602a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.619410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:39.627845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.628848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.629351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.629928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.631038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:39.631549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.633800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.644290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 95/197] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:39.645475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.692884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 103/198] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:39.694212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.770877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa55b7d-6500-4ccb-8b0e-7d79d98982b3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.804247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:39.807740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd935243-7417-4b0d-9410-dbb73585d2cf None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.812524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.813459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.813797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.814082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] PATH_INFO: `/v3/users/1379dfebc09041b888bbf8a77ce59938` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.815413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] RBAC: Authorizing `identity:get_user(user_id=1379dfebc09041b888bbf8a77ce59938)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.816881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.818694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 96/199] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/users/1379dfebc09041b888bbf8a77ce59938 => generated 283 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:39.819467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.848078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.849132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.849571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.849977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] PATH_INFO: `/v3/users/f8fca419d65b408f8c418ceedaebec9e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.851383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] RBAC: Authorizing `identity:get_user(user_id=f8fca419d65b408f8c418ceedaebec9e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.853760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.860332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 104/200] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/users/f8fca419d65b408f8c418ceedaebec9e => generated 289 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:39.861447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:39.891437 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:39.939401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673aa8df-010a-48de-b08e-4c52fa8767b0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.973408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b3d4aa-3395-4e3a-8b4e-34ba5b708412 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:39.978921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:39.979805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:39.980089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:39.980367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:39.981591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:39.983211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:39.985242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 97/201] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:39.986024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:40.012963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:40.013983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:40.014372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:40.014830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:40.055449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:40.092879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:40.094613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:40.104370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a6abd1-43ab-4532-904c-2129e580e79d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:40.141378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:40.152156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:40.152915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:40.153205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:40.153486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:40.208435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 105/202] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:40.209277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:40.234032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:40.235702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:40.282058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:40.320110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9888993-c36a-4c2f-bc62-925616b35ec8 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:40.348094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 98/203] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:40.348918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:40.358465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:40.359152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:40.359442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:40.359724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:40.360683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=1379dfebc09041b888bbf8a77ce59938) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:40.361002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:40.362603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:40.369258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 106/204] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:40 2022] GET /identity/v3/role_assignments?user.id=1379dfebc09041b888bbf8a77ce59938&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:40.370143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:40.379720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:40.468795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e66455-8bbb-455f-8d74-aa2d63c488a8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:40.491083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867bbf6b-28d0-4b70-85dd-5418b1b6851e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:40.510061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:40.511205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:40.511642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:40.512013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:40.513935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:40.515716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:40.525022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 99/205] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:40 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:40.526324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:40.531642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:40.532788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:40.533277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:40.533767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:40.535127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:40.536792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:40.538364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:40.538705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 107/206] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:40 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:40.545149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:40.683279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:40.689853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce753785-6d04-4623-8c9b-631cae415399 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:40.700288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689c6d5-17b8-4ea6-a335-775a06de6a14 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:40.726120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b356f62-a463-4d20-9325-485597b041dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:40.727164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b356f62-a463-4d20-9325-485597b041dd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:40.727455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b356f62-a463-4d20-9325-485597b041dd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:40.727739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b356f62-a463-4d20-9325-485597b041dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:40.728755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b356f62-a463-4d20-9325-485597b041dd None admin] RBAC: Adding query filter params (role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=1379dfebc09041b888bbf8a77ce59938) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:40.729083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b356f62-a463-4d20-9325-485597b041dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:40.731192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b356f62-a463-4d20-9325-485597b041dd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:40.737937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d095b48-65ac-4347-bd77-c40de88113cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:40.738506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 100/207] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:40 2022] GET /identity/v3/role_assignments?user.id=1379dfebc09041b888bbf8a77ce59938&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:40.739390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d095b48-65ac-4347-bd77-c40de88113cc None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:40.739878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d095b48-65ac-4347-bd77-c40de88113cc None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:40.740368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d095b48-65ac-4347-bd77-c40de88113cc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:40.741416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d095b48-65ac-4347-bd77-c40de88113cc None admin] RBAC: Adding query filter params (role.id=43637b70b75a4d028f9ca3ea333735b5, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a, user.id=f8fca419d65b408f8c418ceedaebec9e) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:40.741939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d095b48-65ac-4347-bd77-c40de88113cc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:40.744271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d095b48-65ac-4347-bd77-c40de88113cc None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:40.751791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 108/208] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:40 2022] GET /identity/v3/role_assignments?user.id=f8fca419d65b408f8c418ceedaebec9e&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=43637b70b75a4d028f9ca3ea333735b5 => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:41.414589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeb50ac-9b20-4329-9a46-d35c85effeb5 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:41.414999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeb50ac-9b20-4329-9a46-d35c85effeb5 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:41.415222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeb50ac-9b20-4329-9a46-d35c85effeb5 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:41.416994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 101/209] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:41.422920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac20e19-07ef-4167-a74f-511878ee30a2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:41.423408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac20e19-07ef-4167-a74f-511878ee30a2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:41.423487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac20e19-07ef-4167-a74f-511878ee30a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:41.441523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-eac20e19-07ef-4167-a74f-511878ee30a2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:41.442774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-eac20e19-07ef-4167-a74f-511878ee30a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:41.508141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:41.517363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 109/210] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:41.541855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeb50ac-9b20-4329-9a46-d35c85effeb5 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:41.624082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ef2ab-4ab1-4c2b-ae7b-3d5696f6f2c4 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:41.624454 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ef2ab-4ab1-4c2b-ae7b-3d5696f6f2c4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:41.624638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ef2ab-4ab1-4c2b-ae7b-3d5696f6f2c4 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:41.626279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 110/211] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:41.631276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d2f97-43c3-4c30-b74e-2435f6d44b85 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:41.631568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d2f97-43c3-4c30-b74e-2435f6d44b85 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:41.631851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d2f97-43c3-4c30-b74e-2435f6d44b85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:41.649061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ef1d2f97-43c3-4c30-b74e-2435f6d44b85 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:41.650250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ef1d2f97-43c3-4c30-b74e-2435f6d44b85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:41.652696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeb50ac-9b20-4329-9a46-d35c85effeb5 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:41.697443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:41.698561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:41.698858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:41.699143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:41.713261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:41.722475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 111/212] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:41.728850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:41.729801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:41.730616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:41.732700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 102/213] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:41.739612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d2f97-43c3-4c30-b74e-2435f6d44b85 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:41.750376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:41.774934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:41.862577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d2f97-43c3-4c30-b74e-2435f6d44b85 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:41.882268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d008f008-14da-40ce-a721-d3fd7ba02247 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:41.889625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:41.912190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:41.913753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:41.915125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:41.916033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:41.917784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:41.918704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:41.921561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:41.926795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:41.927560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:41.927884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:41.928672 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:41.930241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:41.931807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:41.932413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 112/214] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:41.937165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 103/215] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:41.939422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:41.942672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:42.052428 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:42.072354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a813604b-d2fe-4d10-9dde-5b792434208a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:42.084900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1213a-97b0-403d-8f44-bb319aa95989 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:42.126368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:42.127101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:42.127398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:42.127676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:42.128071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:42.128426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:42.129968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:42.130724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:42.130866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:42.131583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] PATH_INFO: `/v3/users/1f205dcda1b64051b6d63f14c89a4df5` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:42.157003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] RBAC: Authorizing `identity:get_user(user_id=1f205dcda1b64051b6d63f14c89a4df5)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:42.158979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:42.160940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 104/216] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:41 2022] GET /identity/v3/users/1f205dcda1b64051b6d63f14c89a4df5 => generated 297 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:42.167366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:42.168473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 113/217] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:25:41 2022] POST /identity/v3/users => generated 297 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:25:42.299347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184014c6-2d57-435b-8f28-b30a1e27670e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:42.340922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:42.341939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:42.342403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:42.342997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:42.360761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:42.425513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:42.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:42.541053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 105/218] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:42.546903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:42.596878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:42.661083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafb8df8-28cc-434d-b414-f722c650f7d3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:42.714500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dafff5-c50d-4dbb-9580-dd5a62a6a305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:42.715382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dafff5-c50d-4dbb-9580-dd5a62a6a305 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:42.715668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dafff5-c50d-4dbb-9580-dd5a62a6a305 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:42.715946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dafff5-c50d-4dbb-9580-dd5a62a6a305 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:42.723203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dafff5-c50d-4dbb-9580-dd5a62a6a305 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:42.724770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dafff5-c50d-4dbb-9580-dd5a62a6a305 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:42.727267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 114/219] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:42 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:42.733409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:42.798885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ddca5f-f01b-4d6d-a718-e47abd59b695 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:42.799190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ddca5f-f01b-4d6d-a718-e47abd59b695 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:42.799453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ddca5f-f01b-4d6d-a718-e47abd59b695 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:42.801096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 115/220] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:42.807057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9130ca1-9ea3-4393-b109-40b3861a591e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:42.807346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9130ca1-9ea3-4393-b109-40b3861a591e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:42.807626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9130ca1-9ea3-4393-b109-40b3861a591e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:42.813707 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:42.843564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b9130ca1-9ea3-4393-b109-40b3861a591e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:42.844783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b9130ca1-9ea3-4393-b109-40b3861a591e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:42.873348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f62cb-f44e-4f47-a589-87bee7a6985f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:42.945144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d2145e-c236-4b8c-881a-97908e93ce0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:42.946275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2145e-c236-4b8c-881a-97908e93ce0c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:42.946606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2145e-c236-4b8c-881a-97908e93ce0c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:42.946888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2145e-c236-4b8c-881a-97908e93ce0c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:42.947747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d2145e-c236-4b8c-881a-97908e93ce0c None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=1f205dcda1b64051b6d63f14c89a4df5) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:42.948065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d2145e-c236-4b8c-881a-97908e93ce0c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:42.950059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d2145e-c236-4b8c-881a-97908e93ce0c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:42.958237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 116/221] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:42.966864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 106/222] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:42 2022] GET /identity/v3/role_assignments?user.id=1f205dcda1b64051b6d63f14c89a4df5&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=0d2588012d264232b288e3d89e4579c3 => generated 259 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:43.004131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9130ca1-9ea3-4393-b109-40b3861a591e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:43.074669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:43.077621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849510de-7f79-4439-a368-b519887d5ca3 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.078070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849510de-7f79-4439-a368-b519887d5ca3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.078619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849510de-7f79-4439-a368-b519887d5ca3 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.081351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 107/223] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:43.088485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3dc832-c0be-4885-a904-cc6018bba0d2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.088795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3dc832-c0be-4885-a904-cc6018bba0d2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.089062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3dc832-c0be-4885-a904-cc6018bba0d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.124191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2f3dc832-c0be-4885-a904-cc6018bba0d2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:43.125345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2f3dc832-c0be-4885-a904-cc6018bba0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:43.184886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9130ca1-9ea3-4393-b109-40b3861a591e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:43.242982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:43.244257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.244751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.245255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.258472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 108/224] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:43.267944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:43.289898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:43.290911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:43.291708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:43.292533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 117/225] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:43.300222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3dc832-c0be-4885-a904-cc6018bba0d2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:43.303176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:43.390542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:43.494519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3dc832-c0be-4885-a904-cc6018bba0d2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:43.498780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:43.505327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebb4098-1a1c-4d88-a45d-4ba30dbff4eb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:43.560861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:43.561559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.561846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.562122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.562928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:43.563264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:43.565065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:43.576910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:43.577741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.578030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.578305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.584463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 109/226] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:43.590666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:43.592698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:43.600923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:43.602351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:43.603204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 118/227] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:25:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:43.608551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:43.726373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39967358-3647-4c0f-8003-7ce8860ff106 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:43.728879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:43.766522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:43.768400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.768893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.769414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] PATH_INFO: `/v3/users/1379dfebc09041b888bbf8a77ce59938` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.770893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] RBAC: Authorizing `identity:get_user(user_id=1379dfebc09041b888bbf8a77ce59938)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:43.773412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:43.775455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 110/228] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:43 2022] GET /identity/v3/users/1379dfebc09041b888bbf8a77ce59938 => generated 283 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:43.781928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:43.872102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87028e1-e2d7-4d78-af4e-7168b4b09e5f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:43.899024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14803e7e-623c-43dd-9bed-2ced870f67ae None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:43.916596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:43.917277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.917558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.917888 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.918984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:43.919380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:43.920799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:43.939611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 119/229] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:25:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:43.948027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:43.972164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:43.973037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:43.973321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:43.973595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:43.974828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:43.976303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:43.985286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 111/230] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:43 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:43.990618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:44.001396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:44.042075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:44.142280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44da32-5b55-48e4-9c01-04fc96005f53 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:44.175045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188b8555-422a-4167-8449-c73e590bb47b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:44.213540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:44.214343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:44.214652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:44.214926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] PATH_INFO: `/v3/users/1379dfebc09041b888bbf8a77ce59938` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:44.216780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] RBAC: Authorizing `identity:get_user(user_id=1379dfebc09041b888bbf8a77ce59938)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:44.219106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:44.220924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 120/231] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:43 2022] GET /identity/v3/users/1379dfebc09041b888bbf8a77ce59938 => generated 283 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:44.224529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:44.225564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:44.225745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:44.225975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/1379dfebc09041b888bbf8a77ce59938/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:44.227721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:44.230728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] RBAC: Authorizing `identity:create_grant(project_id=d41de3454f7a4c95a25be612fbe19b07, user_id=1379dfebc09041b888bbf8a77ce59938, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:44.233113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:44.256668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 112/232] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:43 2022] PUT /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/1379dfebc09041b888bbf8a77ce59938/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:25:44.346105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:44.356875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de94bda-72ff-4d9e-bb1f-c4c7d1bb654b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:44.395517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7778319-afc7-46c9-b623-5c6d6fed6d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:44.396660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7778319-afc7-46c9-b623-5c6d6fed6d9c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:44.396951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7778319-afc7-46c9-b623-5c6d6fed6d9c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:44.397229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7778319-afc7-46c9-b623-5c6d6fed6d9c None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:44.399258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7778319-afc7-46c9-b623-5c6d6fed6d9c None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:44.400767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7778319-afc7-46c9-b623-5c6d6fed6d9c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:44.403035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 121/233] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:44 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:44.410532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:44.541242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:44.545690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0510ea-edb9-469c-af0c-6fc081c703e9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:44.595364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520d86d4-810a-44a3-8630-2a0f92fd86aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:44.596338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520d86d4-810a-44a3-8630-2a0f92fd86aa None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:44.596622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520d86d4-810a-44a3-8630-2a0f92fd86aa None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:44.596952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520d86d4-810a-44a3-8630-2a0f92fd86aa None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/1379dfebc09041b888bbf8a77ce59938/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:44.599522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520d86d4-810a-44a3-8630-2a0f92fd86aa None admin] RBAC: Authorizing `identity:create_grant(project_id=d41de3454f7a4c95a25be612fbe19b07, user_id=1379dfebc09041b888bbf8a77ce59938, role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:44.601001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520d86d4-810a-44a3-8630-2a0f92fd86aa None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:44.616839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 113/234] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:44 2022] PUT /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/1379dfebc09041b888bbf8a77ce59938/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:25:44.679093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e0ac9-df0c-4102-98cc-ec148ad12275 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:44.680147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e0ac9-df0c-4102-98cc-ec148ad12275 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:44.680223 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e0ac9-df0c-4102-98cc-ec148ad12275 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:44.681947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 122/235] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:44.688017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b6308-62e5-4fc5-a65a-7a7b53d63819 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:44.688313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b6308-62e5-4fc5-a65a-7a7b53d63819 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:44.688604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b6308-62e5-4fc5-a65a-7a7b53d63819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:44.712886 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-232b6308-62e5-4fc5-a65a-7a7b53d63819 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:44.714149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-232b6308-62e5-4fc5-a65a-7a7b53d63819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:44.811182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 114/236] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:44.838689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e0ac9-df0c-4102-98cc-ec148ad12275 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:44.967142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e0ac9-df0c-4102-98cc-ec148ad12275 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:45.012940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05daf7bb-5de9-4a9c-a07c-b42bcffb636b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:45.013741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05daf7bb-5de9-4a9c-a07c-b42bcffb636b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.014022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05daf7bb-5de9-4a9c-a07c-b42bcffb636b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.014300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05daf7bb-5de9-4a9c-a07c-b42bcffb636b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.016786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05daf7bb-5de9-4a9c-a07c-b42bcffb636b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:45.018604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05daf7bb-5de9-4a9c-a07c-b42bcffb636b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:45.020598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 123/237] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:45.025967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232b6308-62e5-4fc5-a65a-7a7b53d63819 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:45.090902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cf7208-6433-4335-81bc-caeb4b3bb873 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.091198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cf7208-6433-4335-81bc-caeb4b3bb873 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.091384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cf7208-6433-4335-81bc-caeb4b3bb873 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.093021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 124/238] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:45.099700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696896a2-466a-4975-878a-b885a2d16faa None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.099987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696896a2-466a-4975-878a-b885a2d16faa None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.100269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696896a2-466a-4975-878a-b885a2d16faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.119940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-696896a2-466a-4975-878a-b885a2d16faa None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:45.121142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-696896a2-466a-4975-878a-b885a2d16faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:45.144924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:45.190814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:45.202816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232b6308-62e5-4fc5-a65a-7a7b53d63819 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:45.239860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 125/239] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:45.248703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13c2297-0571-433c-b1c9-3b116448bc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:45.249456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13c2297-0571-433c-b1c9-3b116448bc1b None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.249748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13c2297-0571-433c-b1c9-3b116448bc1b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.250026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13c2297-0571-433c-b1c9-3b116448bc1b None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.250778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13c2297-0571-433c-b1c9-3b116448bc1b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:45.252241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13c2297-0571-433c-b1c9-3b116448bc1b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:45.287384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 115/240] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:25:45 2022] POST /identity/v3/users => generated 293 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:45.292249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696896a2-466a-4975-878a-b885a2d16faa None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:45.427268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696896a2-466a-4975-878a-b885a2d16faa None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:45.483688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:45.484531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.484814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.485091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.486399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec004f94-86e3-4b2f-a34c-429022d61040 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.486945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec004f94-86e3-4b2f-a34c-429022d61040 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.487430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec004f94-86e3-4b2f-a34c-429022d61040 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.489379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 116/241] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:25:45.495469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5c527-c7e7-4e44-8015-46353054a4db None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.496113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5c527-c7e7-4e44-8015-46353054a4db None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.496613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5c527-c7e7-4e44-8015-46353054a4db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.501274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:45.510250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:45.511190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:45.511975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:45.514565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 126/242] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:45.520295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:45.535508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f3d5c527-c7e7-4e44-8015-46353054a4db None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:45.536816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f3d5c527-c7e7-4e44-8015-46353054a4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:45.548192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:45.644159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 117/243] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:45.649982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e201050-fb30-4fa0-963d-49adeaa32ad6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:45.674665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5c527-c7e7-4e44-8015-46353054a4db None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:45.704318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:45.705631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.706080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.706613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.707563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] RBAC: Adding query filter params (name=reader) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:45.708078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:45.709652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:45.727170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 127/244] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:45.732995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:45.776662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:45.807286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5c527-c7e7-4e44-8015-46353054a4db None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:45.846084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:45.865488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:45.866146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.866560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.866903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.877069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a13ce7-5ea1-42a8-bb91-becb2c00867c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:45.895529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:45.904510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:45.905426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:45.906233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:45.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 118/245] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:45.917967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:45.919140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:45.919622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:45.920973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:45.921697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] PATH_INFO: `/v3/users/abcc0bf2216b46c1b733488b5ccc5c69` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:45.955836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] RBAC: Authorizing `identity:get_user(user_id=abcc0bf2216b46c1b733488b5ccc5c69)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:45.959170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:45.965009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 128/246] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:45 2022] GET /identity/v3/users/abcc0bf2216b46c1b733488b5ccc5c69 => generated 297 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:45.981126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:46.087472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3269af0-8171-45b7-8633-2b25b4246dd6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:46.126020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:46.127280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:46.127771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:46.128248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:46.129344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:46.129910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:46.131615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:46.133465 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:46.144103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 119/247] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:46.149147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:46.260496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c928c3-63e9-4d0f-9b05-b9e9488f004c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:46.278645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b32bdf0-4fe8-4eb7-9dae-d74cfc610f00 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:46.317632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:46.318989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:46.319469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:46.319940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] PATH_INFO: `/v3/users/1f205dcda1b64051b6d63f14c89a4df5` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:46.321974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] RBAC: Authorizing `identity:get_user(user_id=1f205dcda1b64051b6d63f14c89a4df5)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:46.323781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:46.326186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 120/248] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:46 2022] GET /identity/v3/users/1f205dcda1b64051b6d63f14c89a4df5 => generated 297 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:46.331434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:46.332081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:46.332765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:46.333049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:46.333340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:46.343540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:46.402081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:46.444397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb55ef0-200c-46ac-8a15-d5684728781c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:46.483046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:46.484269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:46.484774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:46.485243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:46.486671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:46.488841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:46.491176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 121/249] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:46 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:46.492349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:46.494377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:46.496623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:46.695945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362f78e-cd8d-462b-96eb-e77d16a38ad5 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:46.713806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 129/250] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 733 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 11 14:25:46.719272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:46.760489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:46.770724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:46.772101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:46.773767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:46.774133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/1f205dcda1b64051b6d63f14c89a4df5/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:46.776713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] RBAC: Authorizing `identity:create_grant(project_id=d41de3454f7a4c95a25be612fbe19b07, user_id=1f205dcda1b64051b6d63f14c89a4df5, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:46.778203 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:46.800934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:46.805232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 122/251] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:46 2022] PUT /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/1f205dcda1b64051b6d63f14c89a4df5/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:25:46.907721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91b618-fde7-4e48-80d2-383135cbcde0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:46.959304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fddf66-ce03-4248-890c-e2a98922747d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:46.960209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fddf66-ce03-4248-890c-e2a98922747d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:46.960503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fddf66-ce03-4248-890c-e2a98922747d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:46.961375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fddf66-ce03-4248-890c-e2a98922747d None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:46.962842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fddf66-ce03-4248-890c-e2a98922747d None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:46.964377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fddf66-ce03-4248-890c-e2a98922747d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:46.966278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 130/252] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:46 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:46.972323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:47.112108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0b5f87-9762-45ff-ac38-d9f4c520f99a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:47.154002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef11f650-15b1-454d-8dba-6a15578a9028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:47.154708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef11f650-15b1-454d-8dba-6a15578a9028 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:47.154998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef11f650-15b1-454d-8dba-6a15578a9028 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:47.155280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef11f650-15b1-454d-8dba-6a15578a9028 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:47.156138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef11f650-15b1-454d-8dba-6a15578a9028 None admin] RBAC: Adding query filter params (role.id=43637b70b75a4d028f9ca3ea333735b5, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=abcc0bf2216b46c1b733488b5ccc5c69) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:47.156461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef11f650-15b1-454d-8dba-6a15578a9028 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:47.157929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef11f650-15b1-454d-8dba-6a15578a9028 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:47.165181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 123/253] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:46 2022] GET /identity/v3/role_assignments?user.id=abcc0bf2216b46c1b733488b5ccc5c69&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=43637b70b75a4d028f9ca3ea333735b5 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:47.336170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36266d4-e742-40a6-9e7d-bc2bace7cf8e None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:47.336550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36266d4-e742-40a6-9e7d-bc2bace7cf8e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:47.336746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36266d4-e742-40a6-9e7d-bc2bace7cf8e None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:47.338135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 131/254] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:47.343610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626af469-e26d-4e22-9017-cc8b8e47a56c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:47.343914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626af469-e26d-4e22-9017-cc8b8e47a56c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:47.344204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626af469-e26d-4e22-9017-cc8b8e47a56c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:47.367541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-626af469-e26d-4e22-9017-cc8b8e47a56c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:47.368774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-626af469-e26d-4e22-9017-cc8b8e47a56c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:47.436316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:47.460439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 124/255] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:47.498613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36266d4-e742-40a6-9e7d-bc2bace7cf8e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:47.650310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36266d4-e742-40a6-9e7d-bc2bace7cf8e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:47.652994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:47.716908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e210074b-93d9-4a48-9553-8e3388f45b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:47.717722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e210074b-93d9-4a48-9553-8e3388f45b3a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:47.718009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e210074b-93d9-4a48-9553-8e3388f45b3a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:47.718289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e210074b-93d9-4a48-9553-8e3388f45b3a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:47.738584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e210074b-93d9-4a48-9553-8e3388f45b3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-e210074b-93d9-4a48-9553-8e3388f45b3a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:47.746529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:47.747419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:47.748398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 132/256] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:25:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:47.754122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626af469-e26d-4e22-9017-cc8b8e47a56c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:47.854252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85dd2f-e2d5-44ab-9bc3-f510226c02af None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:47.854620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85dd2f-e2d5-44ab-9bc3-f510226c02af None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:47.854844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85dd2f-e2d5-44ab-9bc3-f510226c02af None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:47.856239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 133/257] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:47.868084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86f2f5a-fd63-41b4-893b-7ec18d1ddaf2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:47.868379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86f2f5a-fd63-41b4-893b-7ec18d1ddaf2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:47.868656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86f2f5a-fd63-41b4-893b-7ec18d1ddaf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:47.885961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626af469-e26d-4e22-9017-cc8b8e47a56c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:47.939569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e86f2f5a-fd63-41b4-893b-7ec18d1ddaf2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:47.940759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e86f2f5a-fd63-41b4-893b-7ec18d1ddaf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:47.941928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:47.942652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:47.943157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:47.943450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:47.944225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:47.944543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:47.950520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:47.968957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 125/258] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:47.978289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:48.119601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:48.134411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 134/259] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:48.177206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86f2f5a-fd63-41b4-893b-7ec18d1ddaf2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:48.202379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a0959e-6338-42f2-8f02-8d74eb3986b1 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:48.250558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:48.251456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:48.251748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:48.252033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] PATH_INFO: `/v3/users/1379dfebc09041b888bbf8a77ce59938` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:48.254610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] RBAC: Authorizing `identity:get_user(user_id=1379dfebc09041b888bbf8a77ce59938)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:48.256103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:48.258163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 126/260] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:47 2022] GET /identity/v3/users/1379dfebc09041b888bbf8a77ce59938 => generated 283 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:48.264361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:48.389765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160db7cd-190b-45be-879a-b89ed9ca07c8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:48.418797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:48.427466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4939808f-2eef-4cb4-8c4a-fc1bd7ec54ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:48.428146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4939808f-2eef-4cb4-8c4a-fc1bd7ec54ed None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:48.428441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4939808f-2eef-4cb4-8c4a-fc1bd7ec54ed None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:48.428729 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4939808f-2eef-4cb4-8c4a-fc1bd7ec54ed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:48.533751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4939808f-2eef-4cb4-8c4a-fc1bd7ec54ed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:48.534388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:48.535294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4939808f-2eef-4cb4-8c4a-fc1bd7ec54ed None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:48.551382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86f2f5a-fd63-41b4-893b-7ec18d1ddaf2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:48.598975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da663904-12bd-4caa-9322-3e01167b4cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:48.599948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da663904-12bd-4caa-9322-3e01167b4cc9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:48.600245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da663904-12bd-4caa-9322-3e01167b4cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:48.600530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da663904-12bd-4caa-9322-3e01167b4cc9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:48.634399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da663904-12bd-4caa-9322-3e01167b4cc9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-da663904-12bd-4caa-9322-3e01167b4cc9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:48.643182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:48.644136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:48.644943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:48.658546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 135/261] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:25:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 482 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:48.660294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff0f1cc-5b0a-43aa-93be-d8db3ead04ce None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:48.662762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff0f1cc-5b0a-43aa-93be-d8db3ead04ce None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:48.663252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff0f1cc-5b0a-43aa-93be-d8db3ead04ce None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:48.665562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 136/262] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:48.669575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff0f1cc-5b0a-43aa-93be-d8db3ead04ce None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:48.744989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 127/263] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:48.746789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fba79b3-455e-405f-a190-f3d543553b27 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:48.747294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fba79b3-455e-405f-a190-f3d543553b27 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:48.747820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fba79b3-455e-405f-a190-f3d543553b27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:48.767550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1fba79b3-455e-405f-a190-f3d543553b27 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:48.769523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1fba79b3-455e-405f-a190-f3d543553b27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:48.782825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:48.848869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 128/264] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:48.853695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fba79b3-455e-405f-a190-f3d543553b27 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:48.918418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff0f1cc-5b0a-43aa-93be-d8db3ead04ce None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:48.954505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daae49b-998a-4f73-ac07-3c8dc9ef9df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:48.955347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daae49b-998a-4f73-ac07-3c8dc9ef9df8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:48.955640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daae49b-998a-4f73-ac07-3c8dc9ef9df8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:48.955961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daae49b-998a-4f73-ac07-3c8dc9ef9df8 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:48.956761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daae49b-998a-4f73-ac07-3c8dc9ef9df8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:48.957071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daae49b-998a-4f73-ac07-3c8dc9ef9df8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:48.958484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daae49b-998a-4f73-ac07-3c8dc9ef9df8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:48.968066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 137/265] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:25:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:48.969520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185475f-4ba5-4c94-9727-9126982bd68b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:48.969854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185475f-4ba5-4c94-9727-9126982bd68b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:48.970135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185475f-4ba5-4c94-9727-9126982bd68b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:48.971872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 138/266] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:48.972611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8185475f-4ba5-4c94-9727-9126982bd68b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:48.980877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fba79b3-455e-405f-a190-f3d543553b27 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:48.992523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.026688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.028123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.028768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.029252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.031020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.033378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.035791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 129/267] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:48 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:49.037204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.066827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.086719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8185475f-4ba5-4c94-9727-9126982bd68b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.126224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fd7da4-8e5b-4b37-b476-6d4027d13e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.127382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fd7da4-8e5b-4b37-b476-6d4027d13e5c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.128445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fd7da4-8e5b-4b37-b476-6d4027d13e5c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.128445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fd7da4-8e5b-4b37-b476-6d4027d13e5c None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.139880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fd7da4-8e5b-4b37-b476-6d4027d13e5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-23fd7da4-8e5b-4b37-b476-6d4027d13e5c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:49.149080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:49.150161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:49.151188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:49.153272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 139/268] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:49.154702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe90416-993c-47bc-8589-19aaee2378db None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.155099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe90416-993c-47bc-8589-19aaee2378db None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.155758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe90416-993c-47bc-8589-19aaee2378db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.169998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ee3e7e-3431-4c02-8bb9-5071916c6941 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.173558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ffe90416-993c-47bc-8589-19aaee2378db None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:49.174738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ffe90416-993c-47bc-8589-19aaee2378db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:49.214851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.216933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.217411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.217873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] PATH_INFO: `/v3/users/1379dfebc09041b888bbf8a77ce59938` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.219369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] RBAC: Authorizing `identity:get_user(user_id=1379dfebc09041b888bbf8a77ce59938)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.221036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.223376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 130/269] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/users/1379dfebc09041b888bbf8a77ce59938 => generated 283 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:49.224756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.256068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 140/270] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:49.257157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe90416-993c-47bc-8589-19aaee2378db None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.269445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.332529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.357469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06760c9-a47c-4418-b602-329d8d09ae3d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.397850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe90416-993c-47bc-8589-19aaee2378db None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.405371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.406121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.406493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.406844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.407768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=1379dfebc09041b888bbf8a77ce59938) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:49.408406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.409752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.417310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 131/271] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/role_assignments?user.id=1379dfebc09041b888bbf8a77ce59938&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:49.418166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.438738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.439957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.440483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.441004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.441982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:49.442514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.444224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 141/272] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:49.456944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.531884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.540171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9182c8df-027c-4a6b-9861-4efe72b233ae None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.571114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff6e36d-52cd-42e0-b564-51d32ce4981b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.580837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.581794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.582226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.582776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.602131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.613105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.614052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.614415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.614415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.618374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.620061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.622154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 142/273] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:49.623249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.674363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.676189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.761115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971044e-f223-496a-a60f-712ff7e8768d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.798660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a20e685-a26f-4a93-bcc4-d2b996322c49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.800400 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a20e685-a26f-4a93-bcc4-d2b996322c49 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.800690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a20e685-a26f-4a93-bcc4-d2b996322c49 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.800975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a20e685-a26f-4a93-bcc4-d2b996322c49 None admin] PATH_INFO: `/v3/users/1c3af0fa0a3f4373980f6fb856568263` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.805712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.817550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 132/274] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:49.817550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.819258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a20e685-a26f-4a93-bcc4-d2b996322c49 None admin] RBAC: Authorizing `identity:get_user(user_id=1c3af0fa0a3f4373980f6fb856568263)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.821003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a20e685-a26f-4a93-bcc4-d2b996322c49 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.823490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 143/275] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/users/1c3af0fa0a3f4373980f6fb856568263 => generated 293 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:49.824863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b663628a-615e-432c-ac1e-caba4447e1a6 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.825240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b663628a-615e-432c-ac1e-caba4447e1a6 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.825414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b663628a-615e-432c-ac1e-caba4447e1a6 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.826937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 144/276] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:49.827652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b663628a-615e-432c-ac1e-caba4447e1a6 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.864275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:49.922815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d88f46f-2566-443f-bb99-50875010db71 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.937449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b663628a-615e-432c-ac1e-caba4447e1a6 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:49.958003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d651310-32d5-41c5-bee4-9efe65ba20d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.958712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d651310-32d5-41c5-bee4-9efe65ba20d3 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.959000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d651310-32d5-41c5-bee4-9efe65ba20d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.959280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d651310-32d5-41c5-bee4-9efe65ba20d3 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.959993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d651310-32d5-41c5-bee4-9efe65ba20d3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.962047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d651310-32d5-41c5-bee4-9efe65ba20d3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.971388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:49.972252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.972534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.972869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:49.974081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:49.975940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:49.978505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 145/277] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:49.979294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:49.984725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 133/278] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:25:49 2022] POST /identity/v3/users => generated 293 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:49.985925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e4fb0-b296-487a-8493-8fff8f26d4c3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:49.986236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e4fb0-b296-487a-8493-8fff8f26d4c3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:49.986749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e4fb0-b296-487a-8493-8fff8f26d4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.008570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-122e4fb0-b296-487a-8493-8fff8f26d4c3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:50.010192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-122e4fb0-b296-487a-8493-8fff8f26d4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:50.051040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:50.098222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 134/279] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:50.099520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14e481-0682-4dce-be72-f8b2bee0b668 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.099806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14e481-0682-4dce-be72-f8b2bee0b668 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.100084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14e481-0682-4dce-be72-f8b2bee0b668 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.101940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 135/280] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:50.102691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df14e481-0682-4dce-be72-f8b2bee0b668 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:50.116697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a411e4c-e1c1-4572-b83e-f7fc5f119bd7 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:50.149614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:50.166219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20662bc0-e1a2-4b92-8ce0-f630e784d86f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:50.167631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20662bc0-e1a2-4b92-8ce0-f630e784d86f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.167854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20662bc0-e1a2-4b92-8ce0-f630e784d86f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.169218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20662bc0-e1a2-4b92-8ce0-f630e784d86f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.170330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20662bc0-e1a2-4b92-8ce0-f630e784d86f None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, user.id=1c3af0fa0a3f4373980f6fb856568263, scope.system=all) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:50.170945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20662bc0-e1a2-4b92-8ce0-f630e784d86f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:50.174641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20662bc0-e1a2-4b92-8ce0-f630e784d86f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:50.185428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 146/281] 10.222.0.72 () {56 vars in 1308 bytes} [Mon Jul 11 14:25:49 2022] GET /identity/v3/role_assignments?user.id=1c3af0fa0a3f4373980f6fb856568263&scope.system=all&role.id=0d2588012d264232b288e3d89e4579c3 => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:50.187443 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057ac07e-5540-410f-8345-ca95e119a12a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.187769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057ac07e-5540-410f-8345-ca95e119a12a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.188228 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057ac07e-5540-410f-8345-ca95e119a12a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.210004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-057ac07e-5540-410f-8345-ca95e119a12a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:50.211930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-057ac07e-5540-410f-8345-ca95e119a12a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:50.259728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df14e481-0682-4dce-be72-f8b2bee0b668 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:50.302169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 147/282] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:50.306570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057ac07e-5540-410f-8345-ca95e119a12a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:50.311064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:50.311756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.312040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.312321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.313172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] RBAC: Adding query filter params (role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=1379dfebc09041b888bbf8a77ce59938) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:50.313548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:50.315096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:50.323495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 136/283] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/role_assignments?user.id=1379dfebc09041b888bbf8a77ce59938&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 608 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:50.337371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:50.385651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:50.441310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057ac07e-5540-410f-8345-ca95e119a12a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:50.479388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:50.489253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8766f7-45de-463a-9e77-e7e78b730600 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:50.494770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:50.495606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.495893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.496174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.511421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:50.521906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:50.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:50.523717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:50.524308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 148/284] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:50.531805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:50.539276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:50.540103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.542696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.543281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.560879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:50.572594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:50.573551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:50.574339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:50.576188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 137/285] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:50.582189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:50.689178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0278b4-1490-461a-b5b5-ba88ce4419a6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:50.710372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:50.731962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:50.732785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.732966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.733211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.733979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:50.734304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:50.735732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:50.736724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c92c8-c01c-4eb5-8b5e-c998db15d8ab None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:50.745397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 149/286] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:50.768145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:50.793117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:50.794478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.795347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.795837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.796806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:50.797354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:50.799537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:50.810473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 138/287] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:50.820911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:50.855082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:50.908987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905352fc-cdb0-42d1-9ab1-5daa3e972952 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:50.945813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:50.946722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:50.947006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:50.947281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] PATH_INFO: `/v3/users/1f205dcda1b64051b6d63f14c89a4df5` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:50.948483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] RBAC: Authorizing `identity:get_user(user_id=1f205dcda1b64051b6d63f14c89a4df5)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:50.950212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:50.952265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 150/288] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/users/1f205dcda1b64051b6d63f14c89a4df5 => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:50.959257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:51.035430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696df407-d13b-457b-9b96-17d1c39e6508 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:51.035947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:51.089884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6881b8-6891-4899-bd59-f9cfbe771751 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:51.125162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:51.125825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:51.126108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:51.126385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:51.155164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:51.156058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:51.156342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:51.156620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] PATH_INFO: `/v3/users/abcc0bf2216b46c1b733488b5ccc5c69` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:51.157880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] RBAC: Authorizing `identity:get_user(user_id=abcc0bf2216b46c1b733488b5ccc5c69)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:51.159419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:51.161119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 139/289] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/users/abcc0bf2216b46c1b733488b5ccc5c69 => generated 297 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:51.166944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:51.200725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:51.202210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:51.262910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:51.312477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 151/290] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:51.317847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:51.415271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:51.430462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23dcb3d-7353-4d8d-9de1-77c2763d0c3c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:51.452809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3af812-e174-4cae-b2a0-ef72b5c9ad95 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:51.466581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:51.467489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:51.467773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:51.468046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:51.469246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:51.470763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:51.472651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 152/291] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:51 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:51.478662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:51.508441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:51.531002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:51.532243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:51.532790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:51.533272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:51.536722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:51.538450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:51.541464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 140/292] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:51 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 375 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:51.547344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:51.603804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c914a2-0e1d-4a9b-997f-6dd7d3e11ec3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:51.640112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc68469-3a72-45de-9095-5eab43fee6e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:51.640782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc68469-3a72-45de-9095-5eab43fee6e1 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:51.641067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc68469-3a72-45de-9095-5eab43fee6e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:51.641343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc68469-3a72-45de-9095-5eab43fee6e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:51.642197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc68469-3a72-45de-9095-5eab43fee6e1 None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=1f205dcda1b64051b6d63f14c89a4df5) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:51.642579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc68469-3a72-45de-9095-5eab43fee6e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:51.644262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc68469-3a72-45de-9095-5eab43fee6e1 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:51.651558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 153/293] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:51 2022] GET /identity/v3/role_assignments?user.id=1f205dcda1b64051b6d63f14c89a4df5&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=0d2588012d264232b288e3d89e4579c3 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:51.771085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fcacf6-d4c0-4148-9a14-26739a4283c4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:51.823190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ef90a5-622f-4dd8-afe7-ff642d1b80c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:51.824159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ef90a5-622f-4dd8-afe7-ff642d1b80c1 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:51.824699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ef90a5-622f-4dd8-afe7-ff642d1b80c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:51.824997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ef90a5-622f-4dd8-afe7-ff642d1b80c1 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/abcc0bf2216b46c1b733488b5ccc5c69/roles/43637b70b75a4d028f9ca3ea333735b5` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:51.832251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ef90a5-622f-4dd8-afe7-ff642d1b80c1 None admin] RBAC: Authorizing `identity:create_grant(project_id=d41de3454f7a4c95a25be612fbe19b07, user_id=abcc0bf2216b46c1b733488b5ccc5c69, role_id=43637b70b75a4d028f9ca3ea333735b5)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:51.833789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ef90a5-622f-4dd8-afe7-ff642d1b80c1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:51.855092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 141/294] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:25:51 2022] PUT /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07/users/abcc0bf2216b46c1b733488b5ccc5c69/roles/43637b70b75a4d028f9ca3ea333735b5 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:25:52.549729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bfa23c-8a11-4896-95e0-5d19dd74dc4b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:52.550960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bfa23c-8a11-4896-95e0-5d19dd74dc4b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:52.551557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bfa23c-8a11-4896-95e0-5d19dd74dc4b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:52.553472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 154/295] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:52.559108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0764af1-82d9-44bf-b038-88440e0f8f22 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:52.559975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0764af1-82d9-44bf-b038-88440e0f8f22 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:52.560449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0764af1-82d9-44bf-b038-88440e0f8f22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:52.578803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b0764af1-82d9-44bf-b038-88440e0f8f22 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:52.580456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b0764af1-82d9-44bf-b038-88440e0f8f22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:52.590957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:52.665112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 142/296] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:52.689568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bfa23c-8a11-4896-95e0-5d19dd74dc4b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:52.779438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:52.803436 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bfa23c-8a11-4896-95e0-5d19dd74dc4b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:52.843483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809765ef-9eba-4b33-998a-1241dbc74a5d None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:52.844418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809765ef-9eba-4b33-998a-1241dbc74a5d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:52.844544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:52.845196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809765ef-9eba-4b33-998a-1241dbc74a5d None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:52.845366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:52.845573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:52.845966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:52.847512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 143/297] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:52 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:52.853542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d0fc5e-6e56-41f6-b5f8-ac201bc135fd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:52.854106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d0fc5e-6e56-41f6-b5f8-ac201bc135fd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:52.854706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d0fc5e-6e56-41f6-b5f8-ac201bc135fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:52.857506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:52.865758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:52.867433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:52.868242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 155/298] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:52.874715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:52.876686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f8d0fc5e-6e56-41f6-b5f8-ac201bc135fd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:52.878506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f8d0fc5e-6e56-41f6-b5f8-ac201bc135fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:52.953303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 144/299] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:52.954766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050b2ba9-10df-4d22-bba0-54be7f3d9dbe None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:52.955032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050b2ba9-10df-4d22-bba0-54be7f3d9dbe None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:52.955311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050b2ba9-10df-4d22-bba0-54be7f3d9dbe None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:52.959015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 145/300] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:52.962351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284fc00-b553-44c4-b889-c5ad58b5ab4f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:52.962806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284fc00-b553-44c4-b889-c5ad58b5ab4f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:52.963291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284fc00-b553-44c4-b889-c5ad58b5ab4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:52.985262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3284fc00-b553-44c4-b889-c5ad58b5ab4f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:52.986484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3284fc00-b553-44c4-b889-c5ad58b5ab4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:52.993853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6df01c-f54d-4bab-ada0-d5165df16fbc None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.029733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.030411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.030723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.031001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.031755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:53.032066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.033474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.039968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:53.044741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 156/301] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:53.045683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.065563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 146/302] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:53.066537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3284fc00-b553-44c4-b889-c5ad58b5ab4f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.083172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:53.157860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e56e7-d773-4f55-9d35-1ef60cdc7432 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.190575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3284fc00-b553-44c4-b889-c5ad58b5ab4f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.200851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.201665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.202173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.202278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.215803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:53.223893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:53.224946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:53.225756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 157/303] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:53.226561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.235106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.235984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.236271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] PATH_INFO: `/v3/users/39abb52197b74f59baf37ff77d3e3d35` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.252775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] RBAC: Authorizing `identity:get_user(user_id=39abb52197b74f59baf37ff77d3e3d35)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.254271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.261582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 147/304] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:53 2022] GET /identity/v3/users/39abb52197b74f59baf37ff77d3e3d35 => generated 293 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:53.261582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.298242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:53.328614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:53.341750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af583b3-6931-46ef-9ff1-84db1b80c5aa None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.373473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e51455-ef66-4b70-a02e-21b440fe45e2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.375427 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.377139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.377483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.377788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.379150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.386580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 158/305] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:53.387409 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.411740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.412686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.412974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.413258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.414879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:53.414879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.416602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.425396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 148/306] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:53.426255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.509007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae440ca-4ea0-436b-b65b-e74e1c9820db None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.537599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d06c58d-74f1-4fd0-9760-0b44e8c9c166 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.539457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:53.547357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.548018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.548505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.548612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.549403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] RBAC: Adding query filter params (role.id=43637b70b75a4d028f9ca3ea333735b5, user.id=39abb52197b74f59baf37ff77d3e3d35, scope.system=all) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:53.549836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.551214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.566463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 159/307] 10.222.0.72 () {56 vars in 1308 bytes} [Mon Jul 11 14:25:53 2022] GET /identity/v3/role_assignments?user.id=39abb52197b74f59baf37ff77d3e3d35&scope.system=all&role.id=43637b70b75a4d028f9ca3ea333735b5 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:53.566463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.577810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:53.594071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.595393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.595693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.595983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] PATH_INFO: `/v3/groups` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.596711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.598420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.620913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 149/308] 10.222.0.72 () {60 vars in 1134 bytes} [Mon Jul 11 14:25:53 2022] POST /identity/v3/groups => generated 229 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:25:53.692134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412f0a11-56a2-4bce-a698-56c7334a767f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.736671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ded16e-aeff-46e2-a16c-a9f0a63496ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.737661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ded16e-aeff-46e2-a16c-a9f0a63496ab None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.737954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ded16e-aeff-46e2-a16c-a9f0a63496ab None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.738241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ded16e-aeff-46e2-a16c-a9f0a63496ab None admin] PATH_INFO: `/v3/users/1c3af0fa0a3f4373980f6fb856568263` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.739774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ded16e-aeff-46e2-a16c-a9f0a63496ab None admin] RBAC: Authorizing `identity:get_user(user_id=1c3af0fa0a3f4373980f6fb856568263)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.741358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ded16e-aeff-46e2-a16c-a9f0a63496ab None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.744046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 160/309] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:53 2022] GET /identity/v3/users/1c3af0fa0a3f4373980f6fb856568263 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:53.749642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:53.872849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576998b1-2154-4a9b-a4c2-f49523351608 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:53.908659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6ea1ef-356f-4feb-a172-3435fa8bef9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:53.909538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ea1ef-356f-4feb-a172-3435fa8bef9b None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:53.909825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ea1ef-356f-4feb-a172-3435fa8bef9b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:53.910103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ea1ef-356f-4feb-a172-3435fa8bef9b None admin] PATH_INFO: `/v3/system/users/1c3af0fa0a3f4373980f6fb856568263/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:53.911685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ea1ef-356f-4feb-a172-3435fa8bef9b None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=1c3af0fa0a3f4373980f6fb856568263, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:53.913083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ea1ef-356f-4feb-a172-3435fa8bef9b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:53.925039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 150/310] 10.222.0.72 () {58 vars in 1334 bytes} [Mon Jul 11 14:25:53 2022] PUT /identity/v3/system/users/1c3af0fa0a3f4373980f6fb856568263/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:25:54.050798 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1f01d7-4199-491c-ba58-749c33bece58 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:54.051469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1f01d7-4199-491c-ba58-749c33bece58 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:54.051953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1f01d7-4199-491c-ba58-749c33bece58 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:54.053968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 161/311] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:54.064035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22707d30-fa1c-4977-9577-a15437663e04 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:54.064676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22707d30-fa1c-4977-9577-a15437663e04 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:54.065183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22707d30-fa1c-4977-9577-a15437663e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:54.085405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-22707d30-fa1c-4977-9577-a15437663e04 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:54.087064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-22707d30-fa1c-4977-9577-a15437663e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:54.139463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:54.169432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 151/312] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:54.192991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1f01d7-4199-491c-ba58-749c33bece58 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:54.261424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:54.327899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1f01d7-4199-491c-ba58-749c33bece58 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:54.365698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:54.366544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:54.367367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:54.367644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:54.378868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:54.387449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:54.388385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:54.389186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 162/313] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:54.396415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22707d30-fa1c-4977-9577-a15437663e04 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:54.508321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22707d30-fa1c-4977-9577-a15437663e04 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:54.553432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:54.555301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:54.556090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:54.556594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:54.557910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:54.558460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:54.560077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:54.571330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 152/314] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:54.577659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:54.694048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f362eebe-0f76-400d-83e3-7e09cd053f5c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:54.715599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:54.731816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:54.732711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:54.732994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:54.733269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] PATH_INFO: `/v3/users/abcc0bf2216b46c1b733488b5ccc5c69` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:54.734778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] RBAC: Authorizing `identity:get_user(user_id=abcc0bf2216b46c1b733488b5ccc5c69)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:54.736244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:54.738201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 163/315] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:54 2022] GET /identity/v3/users/abcc0bf2216b46c1b733488b5ccc5c69 => generated 297 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:54.743935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:54.795094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:54.867263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb51f6f-e5b4-4c83-9856-6b49e2e42c85 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:54.905039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:54.905711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:54.905999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:54.906277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:54.978545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:54.979965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:55.021447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:55.090064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 153/316] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:55.095863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:55.198137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c49288-61ac-4c1e-a15c-d53ad894e945 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:55.242366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242d5519-f0cb-47f5-b320-c30a05c37ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:55.243277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d5519-f0cb-47f5-b320-c30a05c37ab4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:55.243559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d5519-f0cb-47f5-b320-c30a05c37ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:55.243838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d5519-f0cb-47f5-b320-c30a05c37ab4 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:55.245053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242d5519-f0cb-47f5-b320-c30a05c37ab4 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:55.246561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242d5519-f0cb-47f5-b320-c30a05c37ab4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:55.248611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 164/317] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:25:55 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:55.254289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:55.366025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331db004-f2e9-4cd2-8d42-3a4c06e04cc4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:55.403039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dfe5d7-f146-4e0a-97ca-e7b54045eb83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:55.404129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dfe5d7-f146-4e0a-97ca-e7b54045eb83 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:55.404613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dfe5d7-f146-4e0a-97ca-e7b54045eb83 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:55.405090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dfe5d7-f146-4e0a-97ca-e7b54045eb83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:55.406127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dfe5d7-f146-4e0a-97ca-e7b54045eb83 None admin] RBAC: Adding query filter params (role.id=43637b70b75a4d028f9ca3ea333735b5, scope.project.id=d41de3454f7a4c95a25be612fbe19b07, user.id=abcc0bf2216b46c1b733488b5ccc5c69) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:55.406672 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dfe5d7-f146-4e0a-97ca-e7b54045eb83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:55.408343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dfe5d7-f146-4e0a-97ca-e7b54045eb83 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:55.420668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 154/318] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:25:55 2022] GET /identity/v3/role_assignments?user.id=abcc0bf2216b46c1b733488b5ccc5c69&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=43637b70b75a4d028f9ca3ea333735b5 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:55.734651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18b1de-362e-409d-8ddb-effbefb0eced None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:55.735165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18b1de-362e-409d-8ddb-effbefb0eced None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:55.735269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18b1de-362e-409d-8ddb-effbefb0eced None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:55.736609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 165/319] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:55.742986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc41556d-7e51-422b-a38b-c0a7cf133e91 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:55.743273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc41556d-7e51-422b-a38b-c0a7cf133e91 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:55.743723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc41556d-7e51-422b-a38b-c0a7cf133e91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:55.750730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:55.763056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bc41556d-7e51-422b-a38b-c0a7cf133e91 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:55.764269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bc41556d-7e51-422b-a38b-c0a7cf133e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:55.837809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 155/320] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:55.856258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be66f6-d23b-4113-8db2-4045d5450b0b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:55.856550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be66f6-d23b-4113-8db2-4045d5450b0b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:55.857041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be66f6-d23b-4113-8db2-4045d5450b0b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:55.858491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 166/321] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:55.860033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc41556d-7e51-422b-a38b-c0a7cf133e91 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:55.863900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1b2f53-049a-4173-8ab4-a1a71bd77b4d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:55.864189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1b2f53-049a-4173-8ab4-a1a71bd77b4d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:55.864470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1b2f53-049a-4173-8ab4-a1a71bd77b4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:55.881976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-8b1b2f53-049a-4173-8ab4-a1a71bd77b4d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:55.884028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8b1b2f53-049a-4173-8ab4-a1a71bd77b4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:55.936387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:55.954580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 167/322] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:55.977376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc41556d-7e51-422b-a38b-c0a7cf133e91 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:55.977670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1b2f53-049a-4173-8ab4-a1a71bd77b4d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.014196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.015304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.015593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.015874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.018037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:56.028752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:56.036822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:56.037796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:56.039151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 156/323] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:56.044360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.091015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1b2f53-049a-4173-8ab4-a1a71bd77b4d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:56.138511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.139344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.139627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.139906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.141133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:56.142643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:56.144772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 168/324] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:25:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:56.152234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.157698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5a7874-c55b-4113-921d-32cbc4578958 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:56.201807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc6db46-74b0-45c8-8608-f12c9f063948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.202408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc6db46-74b0-45c8-8608-f12c9f063948 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.202858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc6db46-74b0-45c8-8608-f12c9f063948 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.203152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc6db46-74b0-45c8-8608-f12c9f063948 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.203915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc6db46-74b0-45c8-8608-f12c9f063948 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:56.204228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc6db46-74b0-45c8-8608-f12c9f063948 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:56.205668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc6db46-74b0-45c8-8608-f12c9f063948 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:56.215121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 157/325] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:56.215121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac76830-a67d-4ba1-acba-98979940a3d1 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.215121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac76830-a67d-4ba1-acba-98979940a3d1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.215121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac76830-a67d-4ba1-acba-98979940a3d1 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.220211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:56.220285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 158/326] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:56.220861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac76830-a67d-4ba1-acba-98979940a3d1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.270653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67ed9dc-059a-49ab-887e-f40fff145f34 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:56.284881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:56.307654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86228b50-14e4-43bc-830b-56fcb02c084f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.308332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86228b50-14e4-43bc-830b-56fcb02c084f None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.308608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86228b50-14e4-43bc-830b-56fcb02c084f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.308891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86228b50-14e4-43bc-830b-56fcb02c084f None admin] PATH_INFO: `/v3/groups` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.309591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86228b50-14e4-43bc-830b-56fcb02c084f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:56.311090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86228b50-14e4-43bc-830b-56fcb02c084f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:56.326713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 169/327] 10.222.0.72 () {60 vars in 1134 bytes} [Mon Jul 11 14:25:56 2022] POST /identity/v3/groups => generated 226 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:25:56.327987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150b8a7-25fe-488d-8f2c-a381c3681c0c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.328381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150b8a7-25fe-488d-8f2c-a381c3681c0c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.328584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150b8a7-25fe-488d-8f2c-a381c3681c0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.338682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac76830-a67d-4ba1-acba-98979940a3d1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:56.347804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b150b8a7-25fe-488d-8f2c-a381c3681c0c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:56.349106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b150b8a7-25fe-488d-8f2c-a381c3681c0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:56.382056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.383415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.383756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.384042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] PATH_INFO: `/v3/users/39abb52197b74f59baf37ff77d3e3d35` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.385768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] RBAC: Authorizing `identity:get_user(user_id=39abb52197b74f59baf37ff77d3e3d35)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:56.387045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:56.388843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 159/328] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:56 2022] GET /identity/v3/users/39abb52197b74f59baf37ff77d3e3d35 => generated 293 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:56.394487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.432971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 170/329] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:56.460165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150b8a7-25fe-488d-8f2c-a381c3681c0c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.513892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:56.528385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979719a2-fdd8-4251-ab34-6e8028733083 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:56.556469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:56.569213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.571043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.571539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.571827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] PATH_INFO: `/v3/system/users/39abb52197b74f59baf37ff77d3e3d35/roles/43637b70b75a4d028f9ca3ea333735b5` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.576080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=39abb52197b74f59baf37ff77d3e3d35, role_id=43637b70b75a4d028f9ca3ea333735b5)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:56.577572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:56.588010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 160/330] 10.222.0.72 () {58 vars in 1334 bytes} [Mon Jul 11 14:25:56 2022] PUT /identity/v3/system/users/39abb52197b74f59baf37ff77d3e3d35/roles/43637b70b75a4d028f9ca3ea333735b5 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:25:56.589894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150b8a7-25fe-488d-8f2c-a381c3681c0c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:56.642011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.642875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.643277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.643558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.661405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:56.670651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:56.671617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:56.672414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 171/331] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:56.680566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.798805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eba1dd-50b8-4c99-ad74-61d24645712a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:56.835540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:56.836234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:56.836519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:56.836800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:56.837574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:56.837886 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:56.839318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:56.849820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 161/332] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:56.855944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:56.999247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d30f21-ec7f-4a2d-a9eb-a00ff362fe26 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:57.004169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:57.045767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702f4e86-f95d-4e85-bebc-dac4de6b8bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:57.046700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f4e86-f95d-4e85-bebc-dac4de6b8bcb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:57.046999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f4e86-f95d-4e85-bebc-dac4de6b8bcb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:57.047274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f4e86-f95d-4e85-bebc-dac4de6b8bcb None admin] PATH_INFO: `/v3/users/1c3af0fa0a3f4373980f6fb856568263` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:57.048625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702f4e86-f95d-4e85-bebc-dac4de6b8bcb None admin] RBAC: Authorizing `identity:get_user(user_id=1c3af0fa0a3f4373980f6fb856568263)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:57.050222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702f4e86-f95d-4e85-bebc-dac4de6b8bcb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:57.051877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 172/333] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:25:56 2022] GET /identity/v3/users/1c3af0fa0a3f4373980f6fb856568263 => generated 293 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:57.057715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:57.120273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:57.183913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6c8e36-0d44-4a03-9455-c1ed4074bb92 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:57.223288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6ae43e-3f59-444f-bd3f-ea6df311db97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:57.224310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6ae43e-3f59-444f-bd3f-ea6df311db97 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:57.224592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6ae43e-3f59-444f-bd3f-ea6df311db97 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:57.224867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6ae43e-3f59-444f-bd3f-ea6df311db97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:57.225712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6ae43e-3f59-444f-bd3f-ea6df311db97 None admin] RBAC: Adding query filter params (role.id=0d2588012d264232b288e3d89e4579c3, user.id=1c3af0fa0a3f4373980f6fb856568263, scope.system=all) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:57.226100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6ae43e-3f59-444f-bd3f-ea6df311db97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:57.227605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6ae43e-3f59-444f-bd3f-ea6df311db97 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:57.243296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 162/334] 10.222.0.72 () {56 vars in 1308 bytes} [Mon Jul 11 14:25:57 2022] GET /identity/v3/role_assignments?user.id=1c3af0fa0a3f4373980f6fb856568263&scope.system=all&role.id=0d2588012d264232b288e3d89e4579c3 => generated 510 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:25:58.874989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f1762-ab40-4fbd-99f7-20f4d5153edd None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:58.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f1762-ab40-4fbd-99f7-20f4d5153edd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:58.876492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f1762-ab40-4fbd-99f7-20f4d5153edd None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:58.878300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 173/335] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:58.884130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26251abd-2755-4d8d-9b31-3e5ed5cc043c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:58.884642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26251abd-2755-4d8d-9b31-3e5ed5cc043c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:58.884720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26251abd-2755-4d8d-9b31-3e5ed5cc043c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:58.915446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-26251abd-2755-4d8d-9b31-3e5ed5cc043c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:58.916648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-26251abd-2755-4d8d-9b31-3e5ed5cc043c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:58.997113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 163/336] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:59.029169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f1762-ab40-4fbd-99f7-20f4d5153edd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:59.186522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f1762-ab40-4fbd-99f7-20f4d5153edd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:59.189498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5190141-5fc3-4439-9c40-b8ebaf14bf90 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.189788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5190141-5fc3-4439-9c40-b8ebaf14bf90 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.190070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5190141-5fc3-4439-9c40-b8ebaf14bf90 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.191768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 164/337] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:59.197650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a63b88f-6e0b-457f-b68b-dc4d08512ecf None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.197938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a63b88f-6e0b-457f-b68b-dc4d08512ecf None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.198220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a63b88f-6e0b-457f-b68b-dc4d08512ecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.229156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3a63b88f-6e0b-457f-b68b-dc4d08512ecf None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:59.231639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:59.232482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.232770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.233050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.234717 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3a63b88f-6e0b-457f-b68b-dc4d08512ecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:59.263524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:59.275790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:59.277945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:25:59.279975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 174/338] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:25:59.288425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:59.307488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:59.362809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:59.379800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 165/339] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:25:59.427051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a63b88f-6e0b-457f-b68b-dc4d08512ecf None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:59.480262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a8a9e-3e0e-417d-a999-01bd5eb0f31d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:59.539689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f707123-ac02-4445-b4d1-331ebea93209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:59.540762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f707123-ac02-4445-b4d1-331ebea93209 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.541255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f707123-ac02-4445-b4d1-331ebea93209 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.541736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f707123-ac02-4445-b4d1-331ebea93209 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.542745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f707123-ac02-4445-b4d1-331ebea93209 None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:25:59.543260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f707123-ac02-4445-b4d1-331ebea93209 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:59.545384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f707123-ac02-4445-b4d1-331ebea93209 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:59.555805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 175/340] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:25:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:25:59.557363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7792d62e-9b77-4d26-bac0-c817a3c71fd7 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.557807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7792d62e-9b77-4d26-bac0-c817a3c71fd7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.558236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7792d62e-9b77-4d26-bac0-c817a3c71fd7 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.567346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 176/341] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:25:59 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:25:59.568229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7792d62e-9b77-4d26-bac0-c817a3c71fd7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:59.591610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a63b88f-6e0b-457f-b68b-dc4d08512ecf None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:59.643258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13530c48-78a4-4a7c-b06e-fb46ed1e66b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:59.644416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13530c48-78a4-4a7c-b06e-fb46ed1e66b0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.644923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13530c48-78a4-4a7c-b06e-fb46ed1e66b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.645347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13530c48-78a4-4a7c-b06e-fb46ed1e66b0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.665351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13530c48-78a4-4a7c-b06e-fb46ed1e66b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-13530c48-78a4-4a7c-b06e-fb46ed1e66b0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:25:59.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:25:59.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:25:59.680354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:25:59.680946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 166/342] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:25:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:25:59.681679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c1e8db-8307-4bf8-8298-e85e7f2c2947 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.681971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c1e8db-8307-4bf8-8298-e85e7f2c2947 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.682254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c1e8db-8307-4bf8-8298-e85e7f2c2947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.705148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:59.719100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7792d62e-9b77-4d26-bac0-c817a3c71fd7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:25:59.735617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d6c1e8db-8307-4bf8-8298-e85e7f2c2947 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:25:59.736820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d6c1e8db-8307-4bf8-8298-e85e7f2c2947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:25:59.766859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:25:59.767527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:25:59.767869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:25:59.768150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:25:59.793460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:59.868663 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:25:59.870510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:25:59.900122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 167/343] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:25:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:25:59.901021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c1e8db-8307-4bf8-8298-e85e7f2c2947 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:25:59.974814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:25:59.983444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 177/344] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:25:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:25:59.984792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.096966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0950aa7-c2fb-4648-88a2-81426ec8158f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:00.133826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:00.134668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.134959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.135239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.147137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:00.155802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:00.156753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:00.157560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 178/345] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:25:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:00.158297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c1e8db-8307-4bf8-8298-e85e7f2c2947 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:00.158372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.219758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:00.269806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a62bd3-ab6b-47ee-aeb9-c9c2e03c7868 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:00.270319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:00.271047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.271335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.271620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.272392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:00.273394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:00.274854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:00.284422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 168/346] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:25:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:00.285599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.308614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b06d8d-31eb-4c9a-b4fe-a6a7ac60bb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:00.309913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b06d8d-31eb-4c9a-b4fe-a6a7ac60bb09 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.310535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b06d8d-31eb-4c9a-b4fe-a6a7ac60bb09 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.310830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b06d8d-31eb-4c9a-b4fe-a6a7ac60bb09 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.312349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b06d8d-31eb-4c9a-b4fe-a6a7ac60bb09 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:00.314243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b06d8d-31eb-4c9a-b4fe-a6a7ac60bb09 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:00.316163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:00.317320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 179/347] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:00 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:00.318792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b80431-8295-486c-9e77-df8d3b5a2f08 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.319114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b80431-8295-486c-9e77-df8d3b5a2f08 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.319358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b80431-8295-486c-9e77-df8d3b5a2f08 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.320841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 180/348] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:00.321456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b80431-8295-486c-9e77-df8d3b5a2f08 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.416542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b2eae-650b-457f-924c-5475da3d3427 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:00.439771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b80431-8295-486c-9e77-df8d3b5a2f08 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:00.456834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:00.457575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.457959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.458241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.459418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] RBAC: Adding query filter params (name=reader) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:00.459641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:00.461638 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:00.465418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:00.474257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 169/349] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:26:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:00.475461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.483308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:00.483980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.484266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.484547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.564116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:00.565639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:00.578575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:00.602931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f398d2-0f6e-4efd-a895-a590689f50ad None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:00.683250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8627f3e4-af50-4797-9fcb-1850982c1ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:00.684128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8627f3e4-af50-4797-9fcb-1850982c1ba9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.685015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8627f3e4-af50-4797-9fcb-1850982c1ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.685015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8627f3e4-af50-4797-9fcb-1850982c1ba9 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.695877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8627f3e4-af50-4797-9fcb-1850982c1ba9 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:00.697723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8627f3e4-af50-4797-9fcb-1850982c1ba9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:00.699995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 170/350] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:00 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:00.701000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f864c5-fe49-4975-b258-6699b4d36804 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.701295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f864c5-fe49-4975-b258-6699b4d36804 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.701572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f864c5-fe49-4975-b258-6699b4d36804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.709447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 181/351] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:00.710367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.730885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-34f864c5-fe49-4975-b258-6699b4d36804 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:00.733192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-34f864c5-fe49-4975-b258-6699b4d36804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:00.736126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:00.835296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 171/352] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:00.836512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf26e5-9775-4bea-ab4e-56ef1d2de0de None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.836807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf26e5-9775-4bea-ab4e-56ef1d2de0de None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.837093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf26e5-9775-4bea-ab4e-56ef1d2de0de None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.838765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff2867e-83ce-484e-abe1-b27205e2cab5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:00.839733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 172/353] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:00.840487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabf26e5-9775-4bea-ab4e-56ef1d2de0de None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.884649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:00.885220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:00.885501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:00.885774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] PATH_INFO: `/v3/users/39abb52197b74f59baf37ff77d3e3d35` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:00.888173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] RBAC: Authorizing `identity:get_user(user_id=39abb52197b74f59baf37ff77d3e3d35)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:00.888632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:00.890494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 182/354] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:26:00 2022] GET /identity/v3/users/39abb52197b74f59baf37ff77d3e3d35 => generated 293 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:00.895197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:00.944754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:00.980966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabf26e5-9775-4bea-ab4e-56ef1d2de0de None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.008334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:01.018952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461c1958-128e-495e-a0b8-9d70fe307db5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.028658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dc94a3-b629-4896-b3ba-cd95c4106ab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.029811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc94a3-b629-4896-b3ba-cd95c4106ab8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.030313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc94a3-b629-4896-b3ba-cd95c4106ab8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.030890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc94a3-b629-4896-b3ba-cd95c4106ab8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.031946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dc94a3-b629-4896-b3ba-cd95c4106ab8 None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:01.032486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dc94a3-b629-4896-b3ba-cd95c4106ab8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:01.034490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dc94a3-b629-4896-b3ba-cd95c4106ab8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:01.041450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 173/355] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:00 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=0d2588012d264232b288e3d89e4579c3 => generated 260 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:01.043094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4509eb00-bf1b-4251-b919-cf7f18365bc6 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.043653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4509eb00-bf1b-4251-b919-cf7f18365bc6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.044157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4509eb00-bf1b-4251-b919-cf7f18365bc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.065657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-4509eb00-bf1b-4251-b919-cf7f18365bc6 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:01.067569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4509eb00-bf1b-4251-b919-cf7f18365bc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:01.075845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.076751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.077043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.077324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.078701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:01.080209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:01.086888 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 183/356] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:00 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:01.088241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.169185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 174/357] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:01.170063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4509eb00-bf1b-4251-b919-cf7f18365bc6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.229599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9c975e-1a49-442f-b423-eb79d3d7c480 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.279860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.281291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.281291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.281565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.284554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:01.297789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:01.305845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:01.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 184/358] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:01.310822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.328142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4509eb00-bf1b-4251-b919-cf7f18365bc6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.335965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:01.375037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28b4a80-9ddf-419c-8948-51b63bdaf80b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.375879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28b4a80-9ddf-419c-8948-51b63bdaf80b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.376241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28b4a80-9ddf-419c-8948-51b63bdaf80b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.376524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28b4a80-9ddf-419c-8948-51b63bdaf80b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.377363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28b4a80-9ddf-419c-8948-51b63bdaf80b None admin] RBAC: Adding query filter params (role.id=43637b70b75a4d028f9ca3ea333735b5, user.id=39abb52197b74f59baf37ff77d3e3d35, scope.system=all) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:01.377675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28b4a80-9ddf-419c-8948-51b63bdaf80b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:01.379219 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28b4a80-9ddf-419c-8948-51b63bdaf80b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:01.393929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 175/359] 10.222.0.72 () {56 vars in 1308 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/role_assignments?user.id=39abb52197b74f59baf37ff77d3e3d35&scope.system=all&role.id=43637b70b75a4d028f9ca3ea333735b5 => generated 510 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:01.395377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a5ac86-8fe5-40ee-87f8-3f567b289174 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.395677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a5ac86-8fe5-40ee-87f8-3f567b289174 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.395965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a5ac86-8fe5-40ee-87f8-3f567b289174 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.397294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 176/360] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:01.402260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a5ac86-8fe5-40ee-87f8-3f567b289174 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.448175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d29ee14-f158-4979-88f9-a3a779518687 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.500822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.502112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.502661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.503866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.506040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:01.509177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:01.512153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 185/361] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:01.513602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.549145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a5ac86-8fe5-40ee-87f8-3f567b289174 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.599113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:01.608427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b185c2b-4f1f-4420-ba8f-4498338789fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.609206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b185c2b-4f1f-4420-ba8f-4498338789fe None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.609506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b185c2b-4f1f-4420-ba8f-4498338789fe None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.609796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b185c2b-4f1f-4420-ba8f-4498338789fe None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.625161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b185c2b-4f1f-4420-ba8f-4498338789fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-2b185c2b-4f1f-4420-ba8f-4498338789fe None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:01.634830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:01.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:01.636580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 177/362] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:01.638063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6deaa1c7-75fc-4b2f-aad5-87e1f13e43d1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.638063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6deaa1c7-75fc-4b2f-aad5-87e1f13e43d1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.638580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6deaa1c7-75fc-4b2f-aad5-87e1f13e43d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.650305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:01.657650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6deaa1c7-75fc-4b2f-aad5-87e1f13e43d1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:01.660159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6deaa1c7-75fc-4b2f-aad5-87e1f13e43d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:01.671399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43c2f84-1cda-40d6-9704-3a3d7017882a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.722984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.723777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.724149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.724501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.725324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] RBAC: Adding query filter params (name=member) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:01.725698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:01.727827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:01.738236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 186/363] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:01.739309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.788470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 178/364] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:01.789594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6deaa1c7-75fc-4b2f-aad5-87e1f13e43d1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.890599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66c72d-d76f-431c-9414-7cb09c73a094 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.928434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6deaa1c7-75fc-4b2f-aad5-87e1f13e43d1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:01.939160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.940479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.940479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.940479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.941295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:01.941613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:01.944206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:01.951100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 187/365] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 260 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:01.952109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:01.953007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:01.977467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:01.985722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:01.987586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:01.988127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:01.988642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:01.989672 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:01.991040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:01.993017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:02.007827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 179/366] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:02.014689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:02.093276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b8815-38b2-45b5-9682-3c85f41db13d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:02.132955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:02.133615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:02.133899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:02.134173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:02.157004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1a2723-b7e4-4b09-a4b5-6c8cd4aaff86 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:02.209457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:02.210963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:02.211740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:02.212223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:02.227349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:02.228915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:02.229028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:02.238455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:02.239406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:02.240211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:02.243178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 180/367] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:02.244728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:02.249334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:02.306317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:02.363702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 188/368] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:02.364917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:02.387653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f1e993-382f-4501-a153-23a6f286929c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:02.426031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:02.427049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:02.427541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:02.427961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:02.470371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dff063-9225-40af-a855-ced45b3a0d74 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:02.486012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:02.510105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:02.511042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:02.511735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:02.512020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:02.512300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:02.512641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:02.513061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:02.513375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:02.515455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:02.529715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 189/369] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:02.530482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:02.557961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:02.623219 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 181/370] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:02.624281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:02.640479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3a7b3f-e090-4c12-8fff-3f453b8a90c6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:02.677736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:02.678881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:02.679536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:02.679732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:02.680965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:02.682444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:02.684174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 190/371] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:02 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:02.685032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:02.733945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:02.750785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ef8a7-5687-41fc-b99d-bd68f9ce8284 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:02.797308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:02.798756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:02.799257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:02.800088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:02.817205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:02.829373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99026a47-81ef-4032-8504-66eb2767718c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:02.867813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:02.868806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:02.869455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:02.869455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:02.870786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:02.872610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:02.874855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 191/372] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:02 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:02.875656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:02.911907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:02.915426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.000122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b88ead-0ca1-4cd5-8a7e-2c9d1188cb0a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.013580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:03.047499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 182/373] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:03.048946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.054185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.055190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.055548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.055899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.057179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:03.058784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.061210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 192/374] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:02 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:03.062349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.106595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:03.175466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b826f-2ce5-4d4f-93f4-df0c80480e21 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.183278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cb4d9-0c61-45e7-b3bd-a2e218fe7a34 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.215443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.216666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.217163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.217657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.219104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:03.220779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.222352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.223256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.223478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 183/375] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:03.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.223795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] PATH_INFO: `/v3/projects/e2b4fae2f8fe4791bd2046666efe9390` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.224769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.225012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] RBAC: Authorizing `identity:get_project(project_id=e2b4fae2f8fe4791bd2046666efe9390)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:03.230449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.230449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 193/376] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/projects/e2b4fae2f8fe4791bd2046666efe9390 => generated 344 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:03.231507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.285199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:03.344010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d706133f-6906-46dd-9d28-268016b035a5 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.345604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4c4756-f391-4c4b-92a5-f2a06b8fefaf None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.355397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:03.384699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd923496-9a2e-4be2-9518-8f6ae3de8268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.385359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.385770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd923496-9a2e-4be2-9518-8f6ae3de8268 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.386024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.386311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.386553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd923496-9a2e-4be2-9518-8f6ae3de8268 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.386657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.387049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd923496-9a2e-4be2-9518-8f6ae3de8268 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.387453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=d41de3454f7a4c95a25be612fbe19b07) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:03.387769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:03.388332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd923496-9a2e-4be2-9518-8f6ae3de8268 None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=d41de3454f7a4c95a25be612fbe19b07) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:03.388781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd923496-9a2e-4be2-9518-8f6ae3de8268 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:03.389215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.390518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd923496-9a2e-4be2-9518-8f6ae3de8268 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.395912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 194/377] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 260 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:03.396744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.397172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 184/378] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=0d2588012d264232b288e3d89e4579c3 => generated 260 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc3560-6578-4bb9-a273-13d8eee2d623 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.399145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc3560-6578-4bb9-a273-13d8eee2d623 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.399810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc3560-6578-4bb9-a273-13d8eee2d623 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.407412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 185/379] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:03 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:03.413030 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4417149-0cef-461f-8405-d98dfa290733 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.414373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4417149-0cef-461f-8405-d98dfa290733 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.414855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4417149-0cef-461f-8405-d98dfa290733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.449470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c4417149-0cef-461f-8405-d98dfa290733 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:03.450922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c4417149-0cef-461f-8405-d98dfa290733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:03.599575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 186/380] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:03.602490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e2d69a-fde8-4607-a925-5ff2940958e3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.623377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:03.637209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4417149-0cef-461f-8405-d98dfa290733 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.652662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.653535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.653825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.654103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] PATH_INFO: `/v3/groups/06781ae7b35547a5a22d12e0bff2b82c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.672374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] RBAC: Authorizing `identity:get_group(group_id=06781ae7b35547a5a22d12e0bff2b82c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:03.674178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.676542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 195/381] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/groups/06781ae7b35547a5a22d12e0bff2b82c => generated 229 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:03.684757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.762235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4417149-0cef-461f-8405-d98dfa290733 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.764059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:03.801913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.802942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.803335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.803721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.806005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86fc110-2e59-4e7c-97f8-6cabc1700c1a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:03.821597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:03.825629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:03.826796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:03.827621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:03.833090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 187/382] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:03.834620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:03.867059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:03.868079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:03.868651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:03.869222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:03.870325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] RBAC: Adding query filter params (group.id=06781ae7b35547a5a22d12e0bff2b82c, role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=e2b4fae2f8fe4791bd2046666efe9390) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:03.870960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:03.872797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:03.887677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 196/383] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/role_assignments?group.id=06781ae7b35547a5a22d12e0bff2b82c&scope.project.id=e2b4fae2f8fe4791bd2046666efe9390&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 260 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:03.966876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c6e562-b683-497f-98f7-637507ad40a5 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:04.006962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767c7ecd-ed4d-40ca-a2d8-92e2b75597f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:04.008188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c7ecd-ed4d-40ca-a2d8-92e2b75597f7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:04.008482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c7ecd-ed4d-40ca-a2d8-92e2b75597f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:04.008771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c7ecd-ed4d-40ca-a2d8-92e2b75597f7 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:04.009550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767c7ecd-ed4d-40ca-a2d8-92e2b75597f7 None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:04.009871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767c7ecd-ed4d-40ca-a2d8-92e2b75597f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:04.011302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767c7ecd-ed4d-40ca-a2d8-92e2b75597f7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:04.025861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 188/384] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:26:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:04.027302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:04.112944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:04.214287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60902759-5468-4562-a4b7-3e3062c612b6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:04.303331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:04.304173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:04.304465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:04.304745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:04.306047 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:04.307638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:04.309210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848a14e-d131-426c-8a75-ddc7151b6bb2 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:04.309658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848a14e-d131-426c-8a75-ddc7151b6bb2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:04.309830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848a14e-d131-426c-8a75-ddc7151b6bb2 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:04.311335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 189/385] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:04.314183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 197/386] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:04 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:04.318669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8cbd3c-5a58-4a99-9bd5-f6e5e534236a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:04.318984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8cbd3c-5a58-4a99-9bd5-f6e5e534236a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:04.319258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8cbd3c-5a58-4a99-9bd5-f6e5e534236a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:04.327848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:04.345653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9a8cbd3c-5a58-4a99-9bd5-f6e5e534236a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:04.346915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9a8cbd3c-5a58-4a99-9bd5-f6e5e534236a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:04.383987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:04.467544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9884a3de-a2af-4d79-a16b-72a973da5718 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:04.477363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:04.498465 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 190/387] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:04.521468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:04.523323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:04.523837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:04.524324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:04.528657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:04.530342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:04.532680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 198/388] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:04 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:04.537985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8cbd3c-5a58-4a99-9bd5-f6e5e534236a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:04.540711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:04.697810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37352d7-f4c0-4548-8465-32d392639174 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:04.707394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8cbd3c-5a58-4a99-9bd5-f6e5e534236a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:04.769905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:04.770506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:04.770839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:04.771124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:04.771315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:04.771414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/groups/03da602b88b74ad8b550522f6aa22b8f/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:04.771600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:04.771874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:04.777209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] RBAC: Authorizing `identity:create_grant(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a, group_id=03da602b88b74ad8b550522f6aa22b8f, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:04.778718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:04.792086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:04.805783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:04.806844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:04.808359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:04.809749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 191/389] 10.222.0.72 () {58 vars in 1442 bytes} [Mon Jul 11 14:26:04 2022] PUT /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/groups/03da602b88b74ad8b550522f6aa22b8f/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:04.809930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 199/390] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:04.813202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:04.840156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:04.951841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecfac34-dbcc-46da-b7ce-3c0b0b4c2c32 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:05.003125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:05.003814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:05.004101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:05.004374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:05.005132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:05.005439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:05.006855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:05.023480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 192/391] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:05.029201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:05.073230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:05.146487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f9217-28dc-48ab-b06c-8dbdc65dec22 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:05.184353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:05.185231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:05.185513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:05.185788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:05.187032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:05.188493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:05.190670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 200/392] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:05 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:05.196260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:05.284856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:05.316247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d52a76-96ae-42c5-9447-e3e6c9c7198e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:05.351896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a886277-a60b-4d96-991e-be62d81bac52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:05.352686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a886277-a60b-4d96-991e-be62d81bac52 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:05.352969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a886277-a60b-4d96-991e-be62d81bac52 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:05.353247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a886277-a60b-4d96-991e-be62d81bac52 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:05.354475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a886277-a60b-4d96-991e-be62d81bac52 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:05.355940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a886277-a60b-4d96-991e-be62d81bac52 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:05.358089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 193/393] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:05 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:05.363265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:05.485183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c6f722-eeff-46d7-a3fa-945069ed8be4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:05.486993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:05.521937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7123342f-f64e-41ce-94c5-a41fc3f5324c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:05.522924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7123342f-f64e-41ce-94c5-a41fc3f5324c None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:05.523489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7123342f-f64e-41ce-94c5-a41fc3f5324c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:05.523770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7123342f-f64e-41ce-94c5-a41fc3f5324c None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/groups/03da602b88b74ad8b550522f6aa22b8f/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:05.526003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7123342f-f64e-41ce-94c5-a41fc3f5324c None admin] RBAC: Authorizing `identity:create_grant(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a, group_id=03da602b88b74ad8b550522f6aa22b8f, role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:05.527471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7123342f-f64e-41ce-94c5-a41fc3f5324c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:05.545241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 201/394] 10.222.0.72 () {58 vars in 1442 bytes} [Mon Jul 11 14:26:05 2022] PUT /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a/groups/03da602b88b74ad8b550522f6aa22b8f/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:06.298934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8887160e-adbb-440c-99ba-539734eb4717 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.299330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8887160e-adbb-440c-99ba-539734eb4717 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.299509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8887160e-adbb-440c-99ba-539734eb4717 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.300977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 194/395] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:06.302813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2102114-5290-4ad6-896e-296a6b9a600a None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.303102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2102114-5290-4ad6-896e-296a6b9a600a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.303382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2102114-5290-4ad6-896e-296a6b9a600a None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.305193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 202/396] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:06.308148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5def6924-2f13-4acb-9670-aa5c06185bdb None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.308640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5def6924-2f13-4acb-9670-aa5c06185bdb None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.308958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5def6924-2f13-4acb-9670-aa5c06185bdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.314649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e3a883-b99f-4015-9a16-544049dd9546 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.314943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e3a883-b99f-4015-9a16-544049dd9546 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.315228 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e3a883-b99f-4015-9a16-544049dd9546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.333766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-5def6924-2f13-4acb-9670-aa5c06185bdb None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:06.335238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-5def6924-2f13-4acb-9670-aa5c06185bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:06.348752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-48e3a883-b99f-4015-9a16-544049dd9546 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:06.350020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-48e3a883-b99f-4015-9a16-544049dd9546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:06.469207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 203/397] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:06.469887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 195/398] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:06.480782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142315a0-709e-499d-8c90-4cfa4acb4873 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.481080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142315a0-709e-499d-8c90-4cfa4acb4873 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.481369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142315a0-709e-499d-8c90-4cfa4acb4873 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.483189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 204/399] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:06.488570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1813145-6b4d-4d03-8a46-083afea8620e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.488866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1813145-6b4d-4d03-8a46-083afea8620e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.489152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1813145-6b4d-4d03-8a46-083afea8620e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.491987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142315a0-709e-499d-8c90-4cfa4acb4873 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:06.519102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e1813145-6b4d-4d03-8a46-083afea8620e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:06.520565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e1813145-6b4d-4d03-8a46-083afea8620e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:06.603084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:06.609698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 196/400] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:06.610664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1813145-6b4d-4d03-8a46-083afea8620e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:06.614915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:06.627112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142315a0-709e-499d-8c90-4cfa4acb4873 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:06.683190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:06.684305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.684804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.685287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.698593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:06.708057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:06.709078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:06.709884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:06.712871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 205/401] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:06.713956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:06.751226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1813145-6b4d-4d03-8a46-083afea8620e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:06.789430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:06.790304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.790621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.791656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.802260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:06.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:06.812758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:06.813560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:06.817944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 197/402] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:06.818815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:06.830656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a2c9c-83c1-4462-a71f-65b674d6c374 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:06.867971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:06.869099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.869698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.870197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.881543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:06.887210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:06.891390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:06.892297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:06.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:06.901897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 206/403] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:06.903165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:06.919890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:06.930182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f06c05-4dc5-49d8-809e-3539c08378c6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:06.967895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:06.968565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:06.968847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:06.969121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:06.969993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:06.970316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:06.972150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:06.981876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 198/404] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:26:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:06.986953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.021744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2364dc20-1a8e-4e98-adec-38f8a1973dca None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.059527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.060680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.061169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.061643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.062633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:07.063159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.064967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.075506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 207/405] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.076719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.106193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e528ac6-ae8a-43ca-aa48-5e6bafb97bc8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.134933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:07.144300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.144901 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.145188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.145468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.146275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:07.146620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.148185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.162315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 199/406] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.162315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.171874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:07.201407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff77dda-5117-4225-a18e-bb1a9c656a40 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.238672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.239570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.239887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.240619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.241517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.243020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.249415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 208/407] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:07 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.250312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.279371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6de1aa-9406-4ddf-a8ec-fb9aa1ccb97a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.320657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.321975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.322665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.323165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.324954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.326747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.333428 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 200/408] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:07 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.334616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.361411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7ac687-3be3-4004-82a0-2e08e6f262f7 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.399850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:07.402700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.403907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.404356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.404783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] PATH_INFO: `/v3/groups/06781ae7b35547a5a22d12e0bff2b82c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.406831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] RBAC: Authorizing `identity:get_group(group_id=06781ae7b35547a5a22d12e0bff2b82c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.409431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.416464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 209/409] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:07 2022] GET /identity/v3/groups/06781ae7b35547a5a22d12e0bff2b82c => generated 229 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.417933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.433168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:07.454483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5495c4-f443-4f0e-b78e-520c199e4a95 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.492968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.494050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.494405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.494783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.496176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.497714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.499767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 201/410] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:07 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.500552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.540009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaf32a-3d2b-4c24-9d9d-2b94fb0be791 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.584108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.584995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.586089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.586385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.588132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.589509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.591393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 210/411] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:07 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.592119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.631842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd25479-bdd1-4c88-96f4-2c6c86dec4e0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.658710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:07.677963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f59bf3-de5c-490e-b369-c802e0cf08b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.679483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f59bf3-de5c-490e-b369-c802e0cf08b5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.679775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f59bf3-de5c-490e-b369-c802e0cf08b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.680309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:07.680413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f59bf3-de5c-490e-b369-c802e0cf08b5 None admin] PATH_INFO: `/v3/projects/e2b4fae2f8fe4791bd2046666efe9390` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.681876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f59bf3-de5c-490e-b369-c802e0cf08b5 None admin] RBAC: Authorizing `identity:get_project(project_id=e2b4fae2f8fe4791bd2046666efe9390)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.683844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f59bf3-de5c-490e-b369-c802e0cf08b5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.686286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 202/412] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:07 2022] GET /identity/v3/projects/e2b4fae2f8fe4791bd2046666efe9390 => generated 344 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:07.687709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a710b-5f86-4908-a1bd-65e72d76881f None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.688028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a710b-5f86-4908-a1bd-65e72d76881f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.688585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a710b-5f86-4908-a1bd-65e72d76881f None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.693696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 203/413] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:07.693696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79a710b-5f86-4908-a1bd-65e72d76881f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.711293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89706d76-2a9b-4f8d-9896-6ab3993bc5dd None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.754991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.757242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.757536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.757817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07/groups/03da602b88b74ad8b550522f6aa22b8f/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.759987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] RBAC: Authorizing `identity:create_grant(project_id=d41de3454f7a4c95a25be612fbe19b07, group_id=03da602b88b74ad8b550522f6aa22b8f, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.761440 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.786466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 211/414] 10.222.0.72 () {58 vars in 1442 bytes} [Mon Jul 11 14:26:07 2022] PUT /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07/groups/03da602b88b74ad8b550522f6aa22b8f/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:07.786466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:07.815667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79a710b-5f86-4908-a1bd-65e72d76881f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.855526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f75ae3f-24c4-4920-b8f6-debc6d882a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:07.856539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f75ae3f-24c4-4920-b8f6-debc6d882a01 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.856975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f75ae3f-24c4-4920-b8f6-debc6d882a01 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.857327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f75ae3f-24c4-4920-b8f6-debc6d882a01 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07/groups/03da602b88b74ad8b550522f6aa22b8f/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.859958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f75ae3f-24c4-4920-b8f6-debc6d882a01 None admin] RBAC: Authorizing `identity:create_grant(project_id=d41de3454f7a4c95a25be612fbe19b07, group_id=03da602b88b74ad8b550522f6aa22b8f, role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:07.862034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f75ae3f-24c4-4920-b8f6-debc6d882a01 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:07.882438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 204/415] 10.222.0.72 () {58 vars in 1442 bytes} [Mon Jul 11 14:26:07 2022] PUT /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07/groups/03da602b88b74ad8b550522f6aa22b8f/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:07.882438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ceef6-f734-478e-8ee1-f45424357d3d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:07.882438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ceef6-f734-478e-8ee1-f45424357d3d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:07.882438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ceef6-f734-478e-8ee1-f45424357d3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:07.904700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3f8ceef6-f734-478e-8ee1-f45424357d3d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:07.906308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3f8ceef6-f734-478e-8ee1-f45424357d3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:07.927271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d6fdb-5192-411b-a12b-e2218873420b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:07.972340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:07.990677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 205/416] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:08.010644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ef3bd0-d9ed-4a5d-8401-40bac96ec9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:08.011563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef3bd0-d9ed-4a5d-8401-40bac96ec9b7 None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.011851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef3bd0-d9ed-4a5d-8401-40bac96ec9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.012131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef3bd0-d9ed-4a5d-8401-40bac96ec9b7 None admin] PATH_INFO: `/v3/projects/e2b4fae2f8fe4791bd2046666efe9390/groups/06781ae7b35547a5a22d12e0bff2b82c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.014251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ef3bd0-d9ed-4a5d-8401-40bac96ec9b7 None admin] RBAC: Authorizing `identity:create_grant(project_id=e2b4fae2f8fe4791bd2046666efe9390, group_id=06781ae7b35547a5a22d12e0bff2b82c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:08.015760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ef3bd0-d9ed-4a5d-8401-40bac96ec9b7 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:08.024760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8ceef6-f734-478e-8ee1-f45424357d3d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:08.036069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:08.036069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 212/417] 10.222.0.72 () {58 vars in 1442 bytes} [Mon Jul 11 14:26:07 2022] PUT /identity/v3/projects/e2b4fae2f8fe4791bd2046666efe9390/groups/06781ae7b35547a5a22d12e0bff2b82c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:08.052001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ac8fad-5dfc-41c6-b7bc-cf6768437160 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.052325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ac8fad-5dfc-41c6-b7bc-cf6768437160 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.052570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ac8fad-5dfc-41c6-b7bc-cf6768437160 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.055107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 213/418] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:08.060871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90356562-0628-48bc-8654-521d03357d8f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.061171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90356562-0628-48bc-8654-521d03357d8f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.061449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90356562-0628-48bc-8654-521d03357d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.090171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-90356562-0628-48bc-8654-521d03357d8f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:08.091312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-90356562-0628-48bc-8654-521d03357d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:08.176203 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8ceef6-f734-478e-8ee1-f45424357d3d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:08.186864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 214/419] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:08.208112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90356562-0628-48bc-8654-521d03357d8f None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:08.221004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:08.222483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.222974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.223453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.237335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:08.247259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:08.248233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:08.249016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:08.252369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 206/420] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:08.259409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:08.312789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:08.371083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90356562-0628-48bc-8654-521d03357d8f None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:08.382885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d865e050-960b-4a6e-9a58-df312caf2ea2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:08.414042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:08.428160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:08.429013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.429296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.429569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.437484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:08.439175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.439677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.440170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.441133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:08.441902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:08.442215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:08.443797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:08.451348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:08.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:08.453066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 215/421] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:08.457018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 207/422] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:26:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:08.462116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:08.463229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:08.601705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb7a948-c357-48cd-a946-35de765eb5c9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:08.639930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:08.640991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.641496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.641966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.642955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:08.643463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:08.645105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:08.655163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 216/423] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:08.660989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:08.703999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab2a255-1dd7-46f0-984b-bc7f14c8c490 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:08.706462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:08.726819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:08.755722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:08.756478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.756686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.757502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.777111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8b7831-77b6-4a0c-8da4-a4fc5645461d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:08.813110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:08.814239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:08.814807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:08.815296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:08.892751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:08.894631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:08.947343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:08.949150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:08.986818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:09.007608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 217/424] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:09.014505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:09.123459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d8d7e7-69cd-4a39-969d-4151dd010816 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:09.161050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:09.161478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 208/425] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 698 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:09.168040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:09.171313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:09.171940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:09.172456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:09.173983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:09.175460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:09.182473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:09.184758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 218/426] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:09 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:09.191238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:09.228210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:09.348168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e42b17-33aa-4c82-a22b-08c21897a884 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:09.386270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:09.387256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:09.387536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:09.387806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:09.389053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:09.390558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:09.392554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 219/427] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:09 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:09.398412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:09.432445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd9aae7-00f2-4114-a795-5ae7ad0b4a15 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:09.496518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:09.510643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1937513-37e3-4da4-b6fc-e333f81c61aa None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:09.548974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:09.549769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:09.550124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:09.550495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:09.551414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:09.552082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:09.553611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:09.561727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 220/428] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:09 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 611 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:09.571153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:09.572050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:09.572340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:09.572620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] PATH_INFO: `/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:09.573913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] RBAC: Authorizing `identity:get_project(project_id=20d7b62cf0cc4fd6a3c30e3584353f0a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:09.575736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:09.577848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 209/429] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:09 2022] GET /identity/v3/projects/20d7b62cf0cc4fd6a3c30e3584353f0a => generated 298 bytes in 410 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:09.583489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:09.739519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d36872-a063-4a56-a8e8-e7cffbc3557c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:09.803866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ee992-dfe4-4bd8-8093-f18970be59d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:09.804768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ee992-dfe4-4bd8-8093-f18970be59d1 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:09.805051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ee992-dfe4-4bd8-8093-f18970be59d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:09.805326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ee992-dfe4-4bd8-8093-f18970be59d1 None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:09.807201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ee992-dfe4-4bd8-8093-f18970be59d1 None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:09.809658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ee992-dfe4-4bd8-8093-f18970be59d1 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:09.813134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 221/430] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:09 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:09.820936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:09.907475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:09.945411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f719f0c-b91e-48c1-a8db-a7fae916e42d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:09.989085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caa8930-3341-4bf6-b8cb-355108ad70af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:09.989762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caa8930-3341-4bf6-b8cb-355108ad70af None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:09.990043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caa8930-3341-4bf6-b8cb-355108ad70af None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:09.990315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caa8930-3341-4bf6-b8cb-355108ad70af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:09.991232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caa8930-3341-4bf6-b8cb-355108ad70af None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:09.991587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caa8930-3341-4bf6-b8cb-355108ad70af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:09.993020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caa8930-3341-4bf6-b8cb-355108ad70af None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:09.999674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 210/431] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:09 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=20d7b62cf0cc4fd6a3c30e3584353f0a&role.id=0d2588012d264232b288e3d89e4579c3 => generated 611 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:10.295555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc9fa6c-23cc-483d-a243-518e7ccb6504 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.296148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc9fa6c-23cc-483d-a243-518e7ccb6504 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.296400 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc9fa6c-23cc-483d-a243-518e7ccb6504 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.298047 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 222/432] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:10.304766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952bbd5a-4f3d-43ec-a53d-059c4a682fd8 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.305076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952bbd5a-4f3d-43ec-a53d-059c4a682fd8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.305335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952bbd5a-4f3d-43ec-a53d-059c4a682fd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.324909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-952bbd5a-4f3d-43ec-a53d-059c4a682fd8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:10.326110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-952bbd5a-4f3d-43ec-a53d-059c4a682fd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:10.403175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 211/433] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:10.426739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91415a04-35bd-41b0-9b42-267b5943e02b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.427147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952bbd5a-4f3d-43ec-a53d-059c4a682fd8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:10.427642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91415a04-35bd-41b0-9b42-267b5943e02b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.428130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91415a04-35bd-41b0-9b42-267b5943e02b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.431669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 223/434] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:10.436400 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f43c3-d467-403c-b5ab-91d319c17a4c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.436976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f43c3-d467-403c-b5ab-91d319c17a4c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.437462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f43c3-d467-403c-b5ab-91d319c17a4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.458352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a36f43c3-d467-403c-b5ab-91d319c17a4c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:10.460533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a36f43c3-d467-403c-b5ab-91d319c17a4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:10.475822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:10.508311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:10.552415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952bbd5a-4f3d-43ec-a53d-059c4a682fd8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:10.561763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 224/435] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:10.586999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36f43c3-d467-403c-b5ab-91d319c17a4c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:10.597383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:10.598616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.598616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.598814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.611365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:10.620668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:10.621643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:10.622629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 212/436] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:10.628046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:10.726765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36f43c3-d467-403c-b5ab-91d319c17a4c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:10.757682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bf2b11-ec5f-4f68-8a98-fdc64d6ec787 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:10.762005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9f415c-de65-42c6-ae81-fd754b0c017f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:10.762873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9f415c-de65-42c6-ae81-fd754b0c017f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.763166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9f415c-de65-42c6-ae81-fd754b0c017f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.763448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9f415c-de65-42c6-ae81-fd754b0c017f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.774682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9f415c-de65-42c6-ae81-fd754b0c017f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ec9f415c-de65-42c6-ae81-fd754b0c017f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:10.782965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:10.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:10.785021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 225/437] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:10.787690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:10.787865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea549f-dccf-4f43-ab1f-4c0dfb9e749b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.788670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea549f-dccf-4f43-ab1f-4c0dfb9e749b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.788960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea549f-dccf-4f43-ab1f-4c0dfb9e749b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.790358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 226/438] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:10.791152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea549f-dccf-4f43-ab1f-4c0dfb9e749b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:10.799751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae08d5e9-7234-4dfc-a2c5-5d88e9e3f66e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:10.800473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08d5e9-7234-4dfc-a2c5-5d88e9e3f66e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.800764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08d5e9-7234-4dfc-a2c5-5d88e9e3f66e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.801045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08d5e9-7234-4dfc-a2c5-5d88e9e3f66e None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.801816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae08d5e9-7234-4dfc-a2c5-5d88e9e3f66e None admin] RBAC: Adding query filter params (name=member) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:10.802198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae08d5e9-7234-4dfc-a2c5-5d88e9e3f66e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:10.803112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:10.804543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae08d5e9-7234-4dfc-a2c5-5d88e9e3f66e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:10.818028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 213/439] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:26:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:10.818028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ebff2-a8fe-473e-b9ed-9abedf8df5d6 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.818028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ebff2-a8fe-473e-b9ed-9abedf8df5d6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.818028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ebff2-a8fe-473e-b9ed-9abedf8df5d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.833613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-648ebff2-a8fe-473e-b9ed-9abedf8df5d6 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:10.834856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-648ebff2-a8fe-473e-b9ed-9abedf8df5d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:10.897345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea549f-dccf-4f43-ab1f-4c0dfb9e749b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:10.904230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 214/440] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:10.905103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ebff2-a8fe-473e-b9ed-9abedf8df5d6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:10.936162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:10.936843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:10.937130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:10.937411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:10.938171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:10.938523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:10.939934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:10.948924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 227/441] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:10.950000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.011237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ebff2-a8fe-473e-b9ed-9abedf8df5d6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.027293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.029964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.047041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.047704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.047993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.048270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.059587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142d9cc2-3cb4-4291-906c-3516315254aa None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.095216 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.096021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.096311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.096588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.107686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:11.115529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:11.116660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:11.117634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:11.118592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 228/442] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:11.119573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.120223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.121816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.222624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3a1431-c5a1-408f-adb9-e7b409642106 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.222842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 215/443] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:11.223727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.243054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.255458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.261925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.262614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.262904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.263186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.330009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba3d9d0-4c98-4ff1-84d8-4f87120449fc None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.334116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.335699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.365787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.367012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.367295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.367567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.368314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:11.368618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.370007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.378997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 216/444] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:11.379819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.442354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 229/445] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:11.443248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.463009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.474464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.484076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243f383-341d-42e1-a151-24d9fd2726b6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.517953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.518866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.519156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.519441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.520659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.522255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.524042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 217/446] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:11.524797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.546894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03207f53-32bb-4221-8519-43a42c37ad69 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.581392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.581957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.582239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.582542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.624904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b12b5c9-d139-4231-853e-b53f0ee5b567 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.651236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.652838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.659839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.660797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.661083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.661357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] PATH_INFO: `/v3/projects/d41de3454f7a4c95a25be612fbe19b07` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.662615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] RBAC: Authorizing `identity:get_project(project_id=d41de3454f7a4c95a25be612fbe19b07)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.664107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.665980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 218/447] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/projects/d41de3454f7a4c95a25be612fbe19b07 => generated 302 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:11.666685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.687799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.689117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.769025 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 230/448] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:11.769894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.794707 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d224a33-0bea-4ed1-9523-61eae0b39838 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.830378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.831292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.831655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.831860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.833134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.837419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.837419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 219/449] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:11.837633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.894804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf128a8-914a-4df7-8cc6-e25ad71693f4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:11.927224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.935220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:11.936082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:11.936364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:11.936639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] PATH_INFO: `/v3/groups/03da602b88b74ad8b550522f6aa22b8f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:11.937807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] RBAC: Authorizing `identity:get_group(group_id=03da602b88b74ad8b550522f6aa22b8f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:11.939471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:11.941537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 231/450] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/groups/03da602b88b74ad8b550522f6aa22b8f => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:11.942401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:11.953908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:11.971032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1de4f4e-d396-4acc-b89e-b15c519abd5b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:12.019358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:12.020232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:12.020515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:12.020791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] PATH_INFO: `/v3/projects/e2b4fae2f8fe4791bd2046666efe9390` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:12.022038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] RBAC: Authorizing `identity:get_project(project_id=e2b4fae2f8fe4791bd2046666efe9390)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:12.024081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:12.025811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 220/451] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/projects/e2b4fae2f8fe4791bd2046666efe9390 => generated 344 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:12.026745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:12.056645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1234d6b-e05b-436d-aa34-d975018d8fdb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:12.095916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:12.096505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:12.096789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:12.097060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:12.097905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=0d2588012d264232b288e3d89e4579c3, scope.project.id=d41de3454f7a4c95a25be612fbe19b07) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:12.098216 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:12.099720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:12.106486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 232/452] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:11 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=0d2588012d264232b288e3d89e4579c3 => generated 611 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:12.107296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:12.154856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0accae-153b-410e-b557-cfc4bc412076 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:12.204634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:12.205344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:12.206035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:12.206319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:12.206628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:12.207503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] RBAC: Adding query filter params (group.id=03da602b88b74ad8b550522f6aa22b8f, role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=d41de3454f7a4c95a25be612fbe19b07) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:12.207953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:12.209299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:12.228584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 221/453] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:12 2022] GET /identity/v3/role_assignments?group.id=03da602b88b74ad8b550522f6aa22b8f&scope.project.id=d41de3454f7a4c95a25be612fbe19b07&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 611 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:12.264118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e091b7ea-6d40-4b17-99e2-7c58ed2cdf70 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:12.306192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f835d19-b87f-46fe-b0fe-49c7af235048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:12.307231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f835d19-b87f-46fe-b0fe-49c7af235048 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:12.307516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f835d19-b87f-46fe-b0fe-49c7af235048 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:12.307793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f835d19-b87f-46fe-b0fe-49c7af235048 None admin] PATH_INFO: `/v3/groups/06781ae7b35547a5a22d12e0bff2b82c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:12.309222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f835d19-b87f-46fe-b0fe-49c7af235048 None admin] RBAC: Authorizing `identity:get_group(group_id=06781ae7b35547a5a22d12e0bff2b82c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:12.311262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f835d19-b87f-46fe-b0fe-49c7af235048 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:12.313108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 233/454] 10.222.0.72 () {56 vars in 1180 bytes} [Mon Jul 11 14:26:12 2022] GET /identity/v3/groups/06781ae7b35547a5a22d12e0bff2b82c => generated 229 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:12.320282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:12.343594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:12.442990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e7b0a-7d9d-4d6c-84ea-ce8ba7316aa2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:12.480780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2665a5bd-9a31-4303-9a15-007f4ab41ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:12.481449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665a5bd-9a31-4303-9a15-007f4ab41ac0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:12.481737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665a5bd-9a31-4303-9a15-007f4ab41ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:12.482021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665a5bd-9a31-4303-9a15-007f4ab41ac0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:12.482940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2665a5bd-9a31-4303-9a15-007f4ab41ac0 None admin] RBAC: Adding query filter params (group.id=06781ae7b35547a5a22d12e0bff2b82c, role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=e2b4fae2f8fe4791bd2046666efe9390) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:12.483258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2665a5bd-9a31-4303-9a15-007f4ab41ac0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:12.484728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2665a5bd-9a31-4303-9a15-007f4ab41ac0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:12.491144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 222/455] 10.222.0.72 () {56 vars in 1376 bytes} [Mon Jul 11 14:26:12 2022] GET /identity/v3/role_assignments?group.id=06781ae7b35547a5a22d12e0bff2b82c&scope.project.id=e2b4fae2f8fe4791bd2046666efe9390&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 611 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:15.261267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2f34a7-c0fc-4fcf-80f6-7d0908f0aa6d None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.262251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2f34a7-c0fc-4fcf-80f6-7d0908f0aa6d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.262792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2f34a7-c0fc-4fcf-80f6-7d0908f0aa6d None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.264706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 234/456] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:15.269914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5affc9a-6531-420d-8976-acd483a53ea2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.270856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5affc9a-6531-420d-8976-acd483a53ea2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.271342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5affc9a-6531-420d-8976-acd483a53ea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.278529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62391522-b1ba-49f0-858e-a239a18fbbee None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.278700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62391522-b1ba-49f0-858e-a239a18fbbee None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.278929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62391522-b1ba-49f0-858e-a239a18fbbee None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.280759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 235/457] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:15.292022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd64272-3b6f-4611-8a96-8052611bf499 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.292592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd64272-3b6f-4611-8a96-8052611bf499 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.293065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd64272-3b6f-4611-8a96-8052611bf499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.298228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f5affc9a-6531-420d-8976-acd483a53ea2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:15.298228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f5affc9a-6531-420d-8976-acd483a53ea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:15.326006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-bcd64272-3b6f-4611-8a96-8052611bf499 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:15.328349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bcd64272-3b6f-4611-8a96-8052611bf499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:15.401420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 223/458] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:15.403280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407de6e-b39a-4c56-a7b2-44a7705fe6fe None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.403705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407de6e-b39a-4c56-a7b2-44a7705fe6fe None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.404124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407de6e-b39a-4c56-a7b2-44a7705fe6fe None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.405833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 224/459] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:15.411789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee40b5-2885-4d13-ac91-c878589b92e0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.412369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee40b5-2885-4d13-ac91-c878589b92e0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.412842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee40b5-2885-4d13-ac91-c878589b92e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.429872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 236/460] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:15.431131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd64272-3b6f-4611-8a96-8052611bf499 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:15.447763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6aee40b5-2885-4d13-ac91-c878589b92e0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:15.449106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6aee40b5-2885-4d13-ac91-c878589b92e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:15.452929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:15.502843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:15.525170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 225/461] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:15.526328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee40b5-2885-4d13-ac91-c878589b92e0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:15.563622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd64272-3b6f-4611-8a96-8052611bf499 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:15.602050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:15.603162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.603592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.604013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.615586 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:15.617597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:15.645116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:15.646065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:15.646966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:15.646966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:15.646966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:15.646966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:15.650525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 237/462] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:15.651844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:15.671206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee40b5-2885-4d13-ac91-c878589b92e0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:15.714892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:15.715797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.716150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.716496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.726981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:15.729777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:15.731846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:15.753653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:15.754884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:15.755766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:15.755766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:15.755766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:15.755766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:15.755766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 226/463] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:15.755766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:15.780415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89da2db-401b-4757-af99-228ee2c66abb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:15.795785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:15.826226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfc6e15-bc4d-4d52-83b4-a3fe738d2b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:15.827083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfc6e15-bc4d-4d52-83b4-a3fe738d2b06 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.827365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfc6e15-bc4d-4d52-83b4-a3fe738d2b06 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.827643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfc6e15-bc4d-4d52-83b4-a3fe738d2b06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.840140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfc6e15-bc4d-4d52-83b4-a3fe738d2b06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:15.841713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfc6e15-bc4d-4d52-83b4-a3fe738d2b06 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-acfc6e15-bc4d-4d52-83b4-a3fe738d2b06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:15.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:15.864761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:15.865823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:15.865823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:15.865823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:15.865823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:15.866529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 238/464] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:15.867811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13d36a-c427-4a67-b9e9-a8c8b0fb2496 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.868167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13d36a-c427-4a67-b9e9-a8c8b0fb2496 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.868369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13d36a-c427-4a67-b9e9-a8c8b0fb2496 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.869810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 239/465] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:15.871514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13d36a-c427-4a67-b9e9-a8c8b0fb2496 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:15.883766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03916823-4efd-4732-8f7c-f706b2fc6f96 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:15.923723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:15.924403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:15.924684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:15.924960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:15.925688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:15.925997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:15.927928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:15.946493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 227/466] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:15.946493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:16.012417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13d36a-c427-4a67-b9e9-a8c8b0fb2496 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:16.013606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:16.066272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7665eb-f910-4596-b994-989df16844c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:16.067006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7665eb-f910-4596-b994-989df16844c2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.067363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7665eb-f910-4596-b994-989df16844c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.067805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7665eb-f910-4596-b994-989df16844c2 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.068863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7665eb-f910-4596-b994-989df16844c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:16.068863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7665eb-f910-4596-b994-989df16844c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:16.070316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7665eb-f910-4596-b994-989df16844c2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:16.084855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d018b6-7912-445e-b7d4-12a2a6d5d7f0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:16.094029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 240/467] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:16.094029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293f358-e78b-4178-9751-f243507805e2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.094029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293f358-e78b-4178-9751-f243507805e2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.094029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293f358-e78b-4178-9751-f243507805e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.113434 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2293f358-e78b-4178-9751-f243507805e2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:16.114695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2293f358-e78b-4178-9751-f243507805e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:16.131518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:16.132252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.132543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.132828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.133568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:16.133890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:16.135324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:16.136866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:16.158177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 228/468] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:16.159404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:16.206510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 241/469] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:16.206510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293f358-e78b-4178-9751-f243507805e2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:16.281525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819d83c-3ab4-410b-90dd-eb404cc796df None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:16.299537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:16.332900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:16.334000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.334776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.335352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.336561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:16.338388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:16.361729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293f358-e78b-4178-9751-f243507805e2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:16.383151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 229/470] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:26:16 2022] POST /identity/v3/users => generated 246 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:16.384635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:16.405762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:16.407482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.407980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.408425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.409870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:16.412140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:16.442790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 242/471] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:26:16 2022] POST /identity/v3/users => generated 248 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:16.445987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:16.490498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:16.611492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715c67f8-d81b-48fa-8375-e1ccb9454ab4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:16.685197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957c3645-afde-4534-b845-9325b25eadc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:16.686091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c3645-afde-4534-b845-9325b25eadc8 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.686596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c3645-afde-4534-b845-9325b25eadc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.686878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c3645-afde-4534-b845-9325b25eadc8 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.689000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957c3645-afde-4534-b845-9325b25eadc8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:16.694740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957c3645-afde-4534-b845-9325b25eadc8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:16.703917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f72b86-180a-477c-a71d-6393455e2a0c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:16.727507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 230/472] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:26:16 2022] POST /identity/v3/users => generated 248 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:16.730213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373d5ac-3a65-4edf-a328-149649101a4c None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.730949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373d5ac-3a65-4edf-a328-149649101a4c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.731554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373d5ac-3a65-4edf-a328-149649101a4c None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.733564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 231/473] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:16.743636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35798d3-a038-4a91-84ab-8c0806bc5cf0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.744265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35798d3-a038-4a91-84ab-8c0806bc5cf0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.744853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35798d3-a038-4a91-84ab-8c0806bc5cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.787100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f35798d3-a038-4a91-84ab-8c0806bc5cf0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:16.795851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f35798d3-a038-4a91-84ab-8c0806bc5cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:16.800299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:16.801133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:16.801415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:16.801692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:16.821465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:16.823147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:16.834481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:16.851841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:16.852792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:16.853677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:16.853677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:16.853677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:16.853677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:16.853677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 243/474] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 409 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:16.861914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:16.946500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:16.965742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 232/475] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:16.990857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35798d3-a038-4a91-84ab-8c0806bc5cf0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:16.995510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cfd5a7-6d78-4939-bd1c-b40c750d25f4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:17.036985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:17.038056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:17.038576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:17.039085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:17.040044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:17.040556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:17.042281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:17.058083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 244/476] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:17.064324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:17.124815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35798d3-a038-4a91-84ab-8c0806bc5cf0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:17.168531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:17.169575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:17.170058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:17.170602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:17.171571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:17.173287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:17.181049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.sql.core [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 11 14:26:17.181049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 11 14:26:17.181049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [parameters: {'id': 'be914a62c1654398a32f60f871129ae4', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 11 14:26:17.181049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80423) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 11 14:26:17.200319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4767f212-7a73-4f28-8789-d90e6f545713 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result = self._query(query) Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application conn.query(q) Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result.read() Jul 11 14:26:17.201544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return ref.to_dict() Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 11 14:26:17.202610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application next(self.gen) Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application yield resource Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application next(self.gen) Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.session.rollback() Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.force_reraise() Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise self.value Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application session.commit() Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 11 14:26:17.203506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._prepare_impl() Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.session.flush() Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._flush(objects) Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application compat.raise_( Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application flush_context.execute() Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rec.execute(self) Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 11 14:26:17.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 11 14:26:17.205228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result = self._query(query) Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application conn.query(q) Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result.read() Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 11 14:26:17.206038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [parameters: {'id': 'be914a62c1654398a32f60f871129ae4', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:17.206860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 11 14:26:17.207647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:17.210867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 233/477] 10.222.0.72 () {60 vars in 1131 bytes} [Mon Jul 11 14:26:16 2022] POST /identity/v3/roles => generated 144 bytes in 221 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:17.219477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:17.242532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:17.260128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:17.262335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:17.262650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:17.262934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:17.264174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:17.265106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:17.268562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:17.307502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 245/478] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:26:17 2022] POST /identity/v3/users => generated 249 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:17.390400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42136823-9865-45e0-8d61-fc9c46e2fe46 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:17.441215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb58914-e1bf-42ff-a9f3-093822639b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:17.442053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb58914-e1bf-42ff-a9f3-093822639b45 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:17.442335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb58914-e1bf-42ff-a9f3-093822639b45 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:17.442677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb58914-e1bf-42ff-a9f3-093822639b45 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:17.463612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb58914-e1bf-42ff-a9f3-093822639b45 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-ccb58914-e1bf-42ff-a9f3-093822639b45 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:17.472483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:17.473447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:17.474259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 234/479] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:17.481836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:17.617185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd44ec6-7540-463b-9f01-c927dd00b0f6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:17.655984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99326c37-a1a0-4ddf-8520-43928115eaa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:17.657044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99326c37-a1a0-4ddf-8520-43928115eaa6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:17.657342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99326c37-a1a0-4ddf-8520-43928115eaa6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:17.657618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99326c37-a1a0-4ddf-8520-43928115eaa6 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:17.658923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99326c37-a1a0-4ddf-8520-43928115eaa6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:17.659549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99326c37-a1a0-4ddf-8520-43928115eaa6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:17.661230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99326c37-a1a0-4ddf-8520-43928115eaa6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:17.675424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 246/480] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:19.080531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513c60a7-6c1d-454b-a615-7ceabf14b8b0 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.081575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513c60a7-6c1d-454b-a615-7ceabf14b8b0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.082072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513c60a7-6c1d-454b-a615-7ceabf14b8b0 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.084019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 235/481] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:19.089803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf409a4-7ef9-4227-8bbe-63b1f262b40f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.090810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf409a4-7ef9-4227-8bbe-63b1f262b40f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.091489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf409a4-7ef9-4227-8bbe-63b1f262b40f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.115314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9cf409a4-7ef9-4227-8bbe-63b1f262b40f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:19.117588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9cf409a4-7ef9-4227-8bbe-63b1f262b40f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:19.174610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51587977-06f8-484b-8b35-fa74aac23c80 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.174922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51587977-06f8-484b-8b35-fa74aac23c80 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.175175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51587977-06f8-484b-8b35-fa74aac23c80 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.176777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 236/482] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:19.176901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:19.185293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36df652-b472-40ec-95a0-c2ea13017f64 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.185618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36df652-b472-40ec-95a0-c2ea13017f64 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.185854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36df652-b472-40ec-95a0-c2ea13017f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.220186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b36df652-b472-40ec-95a0-c2ea13017f64 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:19.221757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b36df652-b472-40ec-95a0-c2ea13017f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:19.240302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 247/483] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:19.313413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf409a4-7ef9-4227-8bbe-63b1f262b40f None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:19.338410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 237/484] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:19.378610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36df652-b472-40ec-95a0-c2ea13017f64 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:19.389180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:19.497957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf409a4-7ef9-4227-8bbe-63b1f262b40f None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:19.542967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:19.543781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.544069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.544348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.571154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:19.579324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:19.580320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:19.581128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 248/485] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:19.591774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36df652-b472-40ec-95a0-c2ea13017f64 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:19.603760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:19.664824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f1d03-4865-4b0e-ad4a-05cbaafc1ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:19.665648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f1d03-4865-4b0e-ad4a-05cbaafc1ee3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.665934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f1d03-4865-4b0e-ad4a-05cbaafc1ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.666215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f1d03-4865-4b0e-ad4a-05cbaafc1ee3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.670778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:19.691558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421f1d03-4865-4b0e-ad4a-05cbaafc1ee3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-421f1d03-4865-4b0e-ad4a-05cbaafc1ee3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:19.700433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:19.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:19.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:19.705110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 238/486] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:19.706718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e9b3a-be70-4b5c-bc5a-0adf271df7cd None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.707192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e9b3a-be70-4b5c-bc5a-0adf271df7cd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.707644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e9b3a-be70-4b5c-bc5a-0adf271df7cd None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.709527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 239/487] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:19.714610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e9b3a-be70-4b5c-bc5a-0adf271df7cd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:19.759648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bb244d-02c1-454d-a7ab-90e58daddd4f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:19.792932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:19.804616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1629957-a058-40f6-a710-bf45a0ee6040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:19.806152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1629957-a058-40f6-a710-bf45a0ee6040 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.806676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1629957-a058-40f6-a710-bf45a0ee6040 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.807513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1629957-a058-40f6-a710-bf45a0ee6040 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.808840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1629957-a058-40f6-a710-bf45a0ee6040 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:19.809364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1629957-a058-40f6-a710-bf45a0ee6040 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:19.811640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1629957-a058-40f6-a710-bf45a0ee6040 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:19.825696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 249/488] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:19.827476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28fc39d-aa4c-488a-ae8e-dff63cc0bd9d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.827974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28fc39d-aa4c-488a-ae8e-dff63cc0bd9d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.828451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28fc39d-aa4c-488a-ae8e-dff63cc0bd9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.846259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e9b3a-be70-4b5c-bc5a-0adf271df7cd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:19.853358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a28fc39d-aa4c-488a-ae8e-dff63cc0bd9d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:19.854916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a28fc39d-aa4c-488a-ae8e-dff63cc0bd9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:19.897566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:19.898300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:19.898657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:19.899100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:19.900044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:19.900347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:19.901753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:19.914686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 240/489] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:19.916280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:19.957768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 250/490] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:19.959130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28fc39d-aa4c-488a-ae8e-dff63cc0bd9d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.001583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.048457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8335db-f82b-411f-bc0b-e9c29997bbf4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.079827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.089488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.091474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.091788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28fc39d-aa4c-488a-ae8e-dff63cc0bd9d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.092655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.093264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.107698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.109611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.130477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:20.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:20.132063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:20.132906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:20.132906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:20.132906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:20.132906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:20.133113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.133113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.133113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.135687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 241/491] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:20.136933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.150345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.151986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:20.172283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:20.173214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:20.174089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:20.174089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:20.174089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:20.174089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:20.174816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 251/492] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:20.175581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.257177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8557dcc3-b05d-4a17-a14b-d17bc86c1aa9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.259659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.286013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b281ee-a17f-4b00-b517-bd8cda3feb00 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.293136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.293943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.294242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.294556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.306620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:20.314550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:20.315804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 242/493] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:20.316677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.331037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4de8a2-dd1c-48c1-a8d5-92b153fe6c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.331705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4de8a2-dd1c-48c1-a8d5-92b153fe6c92 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.331987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4de8a2-dd1c-48c1-a8d5-92b153fe6c92 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.332262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4de8a2-dd1c-48c1-a8d5-92b153fe6c92 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.332991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4de8a2-dd1c-48c1-a8d5-92b153fe6c92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:20.333299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4de8a2-dd1c-48c1-a8d5-92b153fe6c92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.334767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4de8a2-dd1c-48c1-a8d5-92b153fe6c92 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.348219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 252/494] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:20.350561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e389bf3-0ba5-4677-8497-bda8da77d665 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.351051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e389bf3-0ba5-4677-8497-bda8da77d665 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.351483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e389bf3-0ba5-4677-8497-bda8da77d665 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.353270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 253/495] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:20.354921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750af7c7-3d45-4dfc-8924-912620a43ecb None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.355264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750af7c7-3d45-4dfc-8924-912620a43ecb None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.355602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750af7c7-3d45-4dfc-8924-912620a43ecb None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.357370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 254/496] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:20.358237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750af7c7-3d45-4dfc-8924-912620a43ecb None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.368158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.436004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2267bddb-4803-445e-ab80-6e0faf215c90 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.474852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.475109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750af7c7-3d45-4dfc-8924-912620a43ecb None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.476216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.476542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.476826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.477643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:20.478136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.479878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.494167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 243/497] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:20.495063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.518710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6159353-5a45-4a7c-acd8-9a1c0b4d68ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.519424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6159353-5a45-4a7c-acd8-9a1c0b4d68ac None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.519712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6159353-5a45-4a7c-acd8-9a1c0b4d68ac None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.519992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6159353-5a45-4a7c-acd8-9a1c0b4d68ac None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.520763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6159353-5a45-4a7c-acd8-9a1c0b4d68ac None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:20.521073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6159353-5a45-4a7c-acd8-9a1c0b4d68ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.522504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6159353-5a45-4a7c-acd8-9a1c0b4d68ac None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.524111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.538669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 255/498] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:20.538669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a36c189-9e60-4039-a755-d1dabba8854c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.538669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a36c189-9e60-4039-a755-d1dabba8854c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.538669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a36c189-9e60-4039-a755-d1dabba8854c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.559717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5a36c189-9e60-4039-a755-d1dabba8854c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:20.560927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5a36c189-9e60-4039-a755-d1dabba8854c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:20.620102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5fd7f-fdb3-4722-b93d-653a67ae44d6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.621651 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.637011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 256/499] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:20.650684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb3baea-ee7d-4396-aacd-b37be1960220 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.650684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb3baea-ee7d-4396-aacd-b37be1960220 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.650684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb3baea-ee7d-4396-aacd-b37be1960220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.662494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fbb3baea-ee7d-4396-aacd-b37be1960220 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:20.662494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fbb3baea-ee7d-4396-aacd-b37be1960220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:20.663099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.663913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.664203 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.664641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.672577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.674060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:20.690702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:20.697870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 244/500] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:20.698837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.741581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 257/501] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:20.742742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb3baea-ee7d-4396-aacd-b37be1960220 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.813532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.823283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b349736-1ec8-42c4-a165-5f0190b2fee9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.863324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.863599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb3baea-ee7d-4396-aacd-b37be1960220 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:20.864147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.864432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.864711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] PATH_INFO: `/v3/users/glance` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.873502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.875014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:20.891524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:20.892541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 245/502] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/users/glance => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:20.898366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:20.909123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:20.909958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:20.910243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:20.910566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:20.922072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:20.923647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:20.928199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:20.946114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:20.947282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:20.948170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:20.948170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:20.948170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:20.948170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:20.948170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 258/503] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:20.948472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.012514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d6e5b3-929c-4cbb-aef5-2b73220856db None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.049714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.050567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.050858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.051449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.060944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae9d876-c4ae-48dc-bdf7-5a8671ec9118 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.062920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:21.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:21.073512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 246/504] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:21.074593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.079499 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.106164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.106848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.107119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.107390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.108233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:21.108544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.110083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.150106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 259/505] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:26:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:21.150881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.177225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.205983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee1a7e2-c404-4907-8b09-004fcbcc8f6b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.247704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.248514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.248806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.249096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.260905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.263239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.275530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8a5477-cc25-44e9-9d86-67dd91f1be13 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:21.281140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:21.282009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:21.283157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:21.283157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:21.283157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:21.283157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:21.283157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 247/506] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:21.285276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.327260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.328408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.329181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.329331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.330918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:21.331476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.334135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.370402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 260/507] 10.222.0.72 () {56 vars in 1137 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:21.371282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.397604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.408557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5968ef-5be3-4f21-82e4-ce449e36331f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.447778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.448450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.448742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.449024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.450019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:21.450333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.451761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.453388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.468083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 248/508] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:21.468933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.486205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d77311-8ecd-4780-9c19-bb5e7cf7eb92 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.529951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.530670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.531061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.531345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.532111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:21.532424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.534133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.543626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 261/509] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:21.544462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.580585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9c7567-4790-4ce9-9a51-f37965422d25 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.617494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.618472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.618762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.638595 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.656652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da94ace5-6ca5-4faa-a0da-abcca537a69d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.687487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.696478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.697944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.701453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.702178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.702482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.702846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.703506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:21.703816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.705210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.717316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 262/510] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:21.718257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.813332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 249/511] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:21.814175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:21.837116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee586ae6-9c36-4ac3-9c5c-a202b130c6cb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.876420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.877086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.877644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.896080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.925474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:21.934127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660ef39a-773d-4086-bdd3-7aca51778ae2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:21.965082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.966706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:21.973877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:21.974724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:21.975012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:21.975281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:21.983857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:21.985344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:22.001001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:22.001962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:22.002925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 250/512] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:22.003498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.080399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 263/513] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:22.081293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.124560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c583ee-da8d-453d-abdf-d4ae53176a40 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.140272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:22.167515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.168315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.168600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.168880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:22.180918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:22.182337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:22.202732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:22.203716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:22.204572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:22.204572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:22.204572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:22.204572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:22.204804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d7e6ff-1edb-41e5-a5d5-6ff684e8b674 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.204875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 251/514] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:22.204875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.216008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:22.246879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.247558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.247846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.248128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:22.248845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:22.250312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.276883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 264/515] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:26:22 2022] POST /identity/v3/users => generated 247 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:22.277775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.352552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cec809-ec2d-4cd1-857c-c5e9d7cfd98c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.395620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.396905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.396905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.397098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:22.413011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:22.414981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.430871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9536fa87-a8f3-4d11-b099-5f8b466f157c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.432874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:22.438485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:22.446863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:22.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:22.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:22.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:22.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:22.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 252/516] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:22.448457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.495446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.496452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.496916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.497424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:22.498525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:22.499069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:22.500538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.524025 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 265/517] 10.222.0.72 () {56 vars in 1137 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:22.524658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.545470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:22.608184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9e881b-c617-48f7-a210-80ff058c7163 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.654477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.655289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.655578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.655859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:22.672428 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:22.674044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.686943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93421760-3bc3-45ff-a45b-f47a70cad46b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:22.698569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:22.702324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:22.717237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:22.717237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:22.717237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:22.717237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:22.717237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 253/518] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:22.717237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.743973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.744647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.745041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.745323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:22.746069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:22.746382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:22.747923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.772680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 266/519] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:22.772680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.799144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:22.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06185dc9-37e2-43d3-88b3-2be88a966a27 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.852157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:22.891155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.892419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.892419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.892419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:22.893170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:22.893483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:22.894920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:22.908033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 254/520] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:22.908734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:22.925772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365f87e0-16db-49ad-9dec-333a4377b566 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:22.967725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:22.968293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:22.968580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:22.968865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.045384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ccce8e-846e-43c6-a96f-09c0655bf924 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.058308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.060770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.093077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:23.096194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.100500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.100500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.100500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.100500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:23.100500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.101237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.120839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 255/521] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:23.120839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.162358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:23.203898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 267/522] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:23.205461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.267001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cb59a5-53e0-4081-8e38-2d8b107dc1e3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.316258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.317226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.317528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.317773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.327860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.330033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:23.346156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:23.347270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:23.348183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 256/523] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/users/neutron => generated 85 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:23.348869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.360055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7607a433-86aa-4a96-b940-833de4a4b595 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.393075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:23.415680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.416883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.417358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.417816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.430235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.432628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:23.453697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:23.454859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 268/524] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:23.456120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.466542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:23.483729 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb758f62-9f1c-4d7d-97ac-d1dbdd38c5be None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.529582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.530541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.530921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.531062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.541151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.543968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:23.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:23.564594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:23.565451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:23.568165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 257/525] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:23.569522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.586022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bad6ce7-9a3e-41aa-93b3-1d4c3ef850fd None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.641725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.643033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.643516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.643992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.661530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.663844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:23.685926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:23.686978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:23.687835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:23.687835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:23.687835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:23.687835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:23.687835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 269/526] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:23.688823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.715044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b015bf73-b6e8-4146-bd80-c15520ed84bd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.724304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:23.749980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.750710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.750998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.751276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.752366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:23.752503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.753962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.793418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 258/527] 10.222.0.72 () {56 vars in 1139 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:23.794190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.808776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ba47f0-ea48-4116-8ef6-9c19abbf462e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.809841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:23.850311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.851509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.851799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.852068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.852867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:23.853424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.854668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.873958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 270/528] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:23.874852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:23.927611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfd24f7-fc90-49e5-b392-7e4b660cb3db None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.964683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:23.965671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:23.966137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:23.966637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:23.967517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:23.967832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:23.969300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:23.975704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:23.985233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7ec62e-8c7f-46b5-af1d-951ad674470b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:23.994788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 259/529] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:23.995724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.024338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.025015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.025305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.025583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.026316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:24.026672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.028043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:24.045093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 271/530] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:24.045093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.074079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:24.116472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8866164-0dc1-4faf-9426-3a6cc10d5fcd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.160519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c6e8c-d805-4d4c-8eed-c40364f6c4b5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.170508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.171402 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.171857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.172286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.200676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.201374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.201668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.201959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.202864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=2165eacee8954dce975ba81ba8ac4232) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:24.203187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.204764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:24.211954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 272/531] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:24 2022] GET /identity/v3/role_assignments?user.id=2165eacee8954dce975ba81ba8ac4232&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:24.212737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.234367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:24.270367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.271902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:24.336878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fa0e4c-c9dd-48ea-9819-ed5bba4be824 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.353687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:24.379297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.380408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.380904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.381979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.383145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=fb6c2e2ddb3b4d488fa25fa8e258fec0) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:24.383950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.385564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:24.397379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 260/533] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:24.397578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 273/533] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:24 2022] GET /identity/v3/role_assignments?user.id=fb6c2e2ddb3b4d488fa25fa8e258fec0&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:24.398167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.404029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.527325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c3955-4ccb-4358-83ee-64603067955c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.547947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:24.570490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e006d6-9639-46ff-94f6-2401be5bb3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.571358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e006d6-9639-46ff-94f6-2401be5bb3e1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.571532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ef734-7e2a-475d-9be0-b9e2e8c6861f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.571628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e006d6-9639-46ff-94f6-2401be5bb3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.571914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e006d6-9639-46ff-94f6-2401be5bb3e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.587558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e006d6-9639-46ff-94f6-2401be5bb3e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.589773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e006d6-9639-46ff-94f6-2401be5bb3e1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-95e006d6-9639-46ff-94f6-2401be5bb3e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:24.610242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:24.611193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:24.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 261/534] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:24.613149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d013f-217c-428b-b395-515c52b7995d None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.613450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d013f-217c-428b-b395-515c52b7995d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.613789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d013f-217c-428b-b395-515c52b7995d None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.616978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 262/535] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:24.618822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51b2e17-4bd1-458d-9e42-ff4964056b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.619809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d013f-217c-428b-b395-515c52b7995d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.621189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51b2e17-4bd1-458d-9e42-ff4964056b8e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.621470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51b2e17-4bd1-458d-9e42-ff4964056b8e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.621746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51b2e17-4bd1-458d-9e42-ff4964056b8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.635496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51b2e17-4bd1-458d-9e42-ff4964056b8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.636991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51b2e17-4bd1-458d-9e42-ff4964056b8e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-f51b2e17-4bd1-458d-9e42-ff4964056b8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:24.665421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:24.666268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:24.666268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:24.666268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:24.666268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:24.666268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 274/536] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:24.667380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0238bb5-25e9-4009-bb78-168507e45d80 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.667748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0238bb5-25e9-4009-bb78-168507e45d80 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.668057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0238bb5-25e9-4009-bb78-168507e45d80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.677205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:24.693175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e0238bb5-25e9-4009-bb78-168507e45d80 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:24.694460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e0238bb5-25e9-4009-bb78-168507e45d80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:24.786819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d013f-217c-428b-b395-515c52b7995d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.794450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 275/537] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:24.794450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0238bb5-25e9-4009-bb78-168507e45d80 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.837907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.838645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.838927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.839335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.840095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:24.840415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.841990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:24.856185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 263/538] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:24.857079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:24.895479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:24.944435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0238bb5-25e9-4009-bb78-168507e45d80 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.957154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:24.971671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d711e5-7aca-4aa0-879a-c28abf80e5bb None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:24.983627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:24.984655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:24.985162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:24.985642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:24.986621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:24.987134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:24.988739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.002113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 276/539] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:25.003601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:25.010321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.011181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.011513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.011797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.024067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:25.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:25.033161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:25.038630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 264/540] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:25.039542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:25.120356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f362ec8a-13df-467a-aecb-425d8d922227 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:25.145528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:25.150118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f758ed0-8f4e-44d3-b1de-2c4f05b2d135 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:25.160563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.161613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.162106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.163161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.164229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=4f1b0819d52d481898b354a5ae182746) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:25.164765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:25.166507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.173701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 277/541] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/role_assignments?user.id=4f1b0819d52d481898b354a5ae182746&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:25.175007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:25.190485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.191416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.191712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.191980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.200990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:25.202511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:25.218577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:25.219507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:25.220350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 265/542] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:25.231651 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:25.254243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:25.306851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483485af-2cda-4554-815a-4e91dde2758d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:25.347944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.349005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.349500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.350018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.351118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:25.351651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:25.353312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.363875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 278/543] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:25.370067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:25.436916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9ace42-ce73-4fd2-b8af-32a8204dffb2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:25.483066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.484577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:25.485560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.486165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.486756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.488994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:25.489588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:25.491887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.508147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 266/544] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:25.514315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:25.600439 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b8632-a951-46c5-bac9-2bdf0e5883b0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:25.642612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.644077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.644728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.645196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.658197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:25.660327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:25.680790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:25.681623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:25.682486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:25.682486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:25.682486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:25.682486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:25.685114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 279/545] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:25.691826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:25.709195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:25.769670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61758d49-dc05-4231-8d95-fce9bad396a9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:25.834005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.835309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.835578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.835854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.836719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=ac5254fb5eff4108be9ac3992e05a99c) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:25.837038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:25.838595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.845262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 267/546] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/role_assignments?user.id=ac5254fb5eff4108be9ac3992e05a99c&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 259 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:25.887236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:25.892756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539388b7-b47d-4bf6-ac25-3b94dbf04514 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:25.957000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:25.957610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:25.957893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:25.958168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:25.958949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:25.959259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:25.960766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:25.981580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 280/547] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:25.986924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:26.137929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da7ca67-cc5f-4a7d-8d3c-1be3f5523dc0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:26.175031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:26.181088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:26.187472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:26.187760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:26.188218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:26.197499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:26.198989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:26.216680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:26.217614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:26.219148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 268/548] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:25 2022] GET /identity/v3/users/swift => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:26.223441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:26.384065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921b7c5-fe4d-4abf-9ad6-e6657609679a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:26.436467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df3369a-4586-4007-a6c1-77b0c2be67f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:26.437153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df3369a-4586-4007-a6c1-77b0c2be67f8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:26.437436 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df3369a-4586-4007-a6c1-77b0c2be67f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:26.437711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df3369a-4586-4007-a6c1-77b0c2be67f8 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:26.438586 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df3369a-4586-4007-a6c1-77b0c2be67f8 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:26.438899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df3369a-4586-4007-a6c1-77b0c2be67f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:26.440354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df3369a-4586-4007-a6c1-77b0c2be67f8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:26.474411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 281/549] 10.222.0.72 () {56 vars in 1135 bytes} [Mon Jul 11 14:26:26 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:26.480199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:26.625744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7d492e-fd70-4b79-9203-fdf300ea2435 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:26.692789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:26.693457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:26.693746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:26.694584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:26.696218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b6ca77-e49a-463e-8cd0-a1afbfaf5cda None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:26.696510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b6ca77-e49a-463e-8cd0-a1afbfaf5cda None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:26.696790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b6ca77-e49a-463e-8cd0-a1afbfaf5cda None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:26.698233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 282/550] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:26.708294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87269ba4-b956-4e35-95c4-48e3008806d6 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:26.708594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87269ba4-b956-4e35-95c4-48e3008806d6 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:26.708878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87269ba4-b956-4e35-95c4-48e3008806d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:26.739780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-87269ba4-b956-4e35-95c4-48e3008806d6 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:26.741062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-87269ba4-b956-4e35-95c4-48e3008806d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:26.768122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:26.786568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:26.798063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:26.799625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:26.877310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 283/551] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:26.879144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68562c-4859-4d8c-8be5-c06cba282b7b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:26.880438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68562c-4859-4d8c-8be5-c06cba282b7b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:26.881082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68562c-4859-4d8c-8be5-c06cba282b7b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:26.887418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 284/552] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:26 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:26.892790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73f41b-f159-4f58-be98-580ef7fc90f2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:26.893143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73f41b-f159-4f58-be98-580ef7fc90f2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:26.893455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73f41b-f159-4f58-be98-580ef7fc90f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:26.917273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-da73f41b-f159-4f58-be98-580ef7fc90f2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:26.918500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-da73f41b-f159-4f58-be98-580ef7fc90f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:26.972367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 269/553] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:26.973636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:26.994989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 285/554] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:26.995872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da73f41b-f159-4f58-be98-580ef7fc90f2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.072994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.079084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.083848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b742aa57-1446-4aa9-90a9-305f76c7a5ce None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.105545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da73f41b-f159-4f58-be98-580ef7fc90f2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.121282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.122469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.123094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.123583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.134993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:27.143186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:27.144116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:27.144940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:27.145587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.145587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.145587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.145587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.148598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 270/555] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:27.149675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.162351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.164046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:27.184212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:27.185368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:27.186301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:27.186301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:27.186301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:27.186301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:27.186301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 286/556] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:27.187061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.264301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf15f6-ff56-403d-bd4a-583d997b2fa1 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.308223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.309312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.309600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.309879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.311733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8690de4d-cce2-4ad8-8f4f-a83fc0b1ea83 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.325447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:27.336478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:27.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:27.338892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 271/557] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:27.339574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.349372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.360850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.362943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.363879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.364524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.364844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.366340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:27.366578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.367965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.386548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 287/558] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:27.388457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.473825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22c94d5-fc37-4699-af4d-b305331830a7 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.517002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.519180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.520141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.520617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.521608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:27.522171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.522277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae439d-530e-455d-bc20-4c75b44184e9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.524480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.543565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 272/559] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:27.544849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.564755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.565551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.565910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.566257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.567170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:27.567579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.569157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.580131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 288/560] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:27.581196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.649122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.665585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.710302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c67afb2-2701-4ad0-98c4-d1e09e022e2f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.716171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12c0b4-ea63-4b3c-b3b8-10e65d0182ed None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.747621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.748554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.748889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.749167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.756386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.756983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97378a9-f728-422c-98d3-1672fca36733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.757784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97378a9-f728-422c-98d3-1672fca36733 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.757882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.758070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97378a9-f728-422c-98d3-1672fca36733 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.758350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97378a9-f728-422c-98d3-1672fca36733 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.771244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97378a9-f728-422c-98d3-1672fca36733 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.773181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97378a9-f728-422c-98d3-1672fca36733 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:27.773414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:27.774243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:27.775204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:27.779379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 289/561] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:27.780243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d97378a9-f728-422c-98d3-1672fca36733 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:27.794929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:27.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:27.796793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:27.796793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:27.796793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:27.796793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:27.796793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 273/562] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:27.797731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66bc84a-086e-4070-b669-53a2733adda8 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.797731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66bc84a-086e-4070-b669-53a2733adda8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.797963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66bc84a-086e-4070-b669-53a2733adda8 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.799498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 274/563] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:27.800275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66bc84a-086e-4070-b669-53a2733adda8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.901137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7e5d4a-57fa-488e-a47f-3e07a77c3363 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.917602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66bc84a-086e-4070-b669-53a2733adda8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:27.918566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.941255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff7428b-57fd-44a7-b118-601983069d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.942215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff7428b-57fd-44a7-b118-601983069d43 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.942532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff7428b-57fd-44a7-b118-601983069d43 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.942812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff7428b-57fd-44a7-b118-601983069d43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.944932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:27.954893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff7428b-57fd-44a7-b118-601983069d43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.956488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff7428b-57fd-44a7-b118-601983069d43 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.961817 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:27.962526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.962821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.963119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:27.963913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:27.964724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:27.966640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-7ff7428b-57fd-44a7-b118-601983069d43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:27.978823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:27.979825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:27.980724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:27.980724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:27.980724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:27.980724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:27.984798 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 290/564] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:27.985355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 275/565] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:27.985978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415728a8-bc85-426e-87c3-3d54bd2d0ca4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:27.986099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:27.986272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415728a8-bc85-426e-87c3-3d54bd2d0ca4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:27.986587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415728a8-bc85-426e-87c3-3d54bd2d0ca4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.004277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-415728a8-bc85-426e-87c3-3d54bd2d0ca4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:28.005487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-415728a8-bc85-426e-87c3-3d54bd2d0ca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:28.081584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 291/566] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:28.082941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415728a8-bc85-426e-87c3-3d54bd2d0ca4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.108441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7606e82f-0c94-44d7-9a20-83e1f051f85b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.146727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.147703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.148050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.148267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.149278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:28.149605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.151039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.165940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 276/567] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:28.166839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.212076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:28.220949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415728a8-bc85-426e-87c3-3d54bd2d0ca4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.222092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:28.256187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.257380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.257665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.257941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.258708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:28.259014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.260420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.273389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 292/568] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:28.274243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.281595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e44fdca-7c6c-436d-99bb-b5102e778db3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.316834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.317496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.317782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.318058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.319541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=055512e4e3744833ac6a9e61d22f0b63) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:28.319868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.321327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.327254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 277/569] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/role_assignments?user.id=055512e4e3744833ac6a9e61d22f0b63&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:28.328089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.387040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01700ad9-f7fe-4044-905a-3e506df4136c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.428698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6adf6c-7e90-44e5-937e-84052b4c1aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.429738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6adf6c-7e90-44e5-937e-84052b4c1aa7 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.430270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6adf6c-7e90-44e5-937e-84052b4c1aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.430657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6adf6c-7e90-44e5-937e-84052b4c1aa7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.448332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6adf6c-7e90-44e5-937e-84052b4c1aa7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:28.456611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d07e6-85e9-490e-9da5-10f4348e775b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-6a6adf6c-7e90-44e5-937e-84052b4c1aa7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:28.456825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:28.458930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:28.459987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 293/570] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:28.461226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c42d2-2a79-4548-a7e4-e81fdd4c80d9 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.461398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c42d2-2a79-4548-a7e4-e81fdd4c80d9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.461604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c42d2-2a79-4548-a7e4-e81fdd4c80d9 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.466727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 294/571] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:28.466727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c42d2-2a79-4548-a7e4-e81fdd4c80d9 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.478206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:28.486405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:28.515190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.516087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.516744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.516744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.525169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.526694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:28.542910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:28.543940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 278/572] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:28.544962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.600285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c42d2-2a79-4548-a7e4-e81fdd4c80d9 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.639598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4d9c27-8f89-45c2-a3d7-99280dcc0f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.640328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4d9c27-8f89-45c2-a3d7-99280dcc0f67 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.640612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4d9c27-8f89-45c2-a3d7-99280dcc0f67 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.640888 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4d9c27-8f89-45c2-a3d7-99280dcc0f67 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.650352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4d9c27-8f89-45c2-a3d7-99280dcc0f67 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.654174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4d9c27-8f89-45c2-a3d7-99280dcc0f67 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.671193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cb35ef-5094-4a86-aa8f-4dbd529eeae3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-6a4d9c27-8f89-45c2-a3d7-99280dcc0f67 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:28.679611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:28.680630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:28.681588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 295/573] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/users/glance => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:28.683219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cfecd-44f0-4f64-a182-1f3c9f174ece None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.683219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cfecd-44f0-4f64-a182-1f3c9f174ece None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.683219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cfecd-44f0-4f64-a182-1f3c9f174ece None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.700471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ea8cfecd-44f0-4f64-a182-1f3c9f174ece None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:28.701689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ea8cfecd-44f0-4f64-a182-1f3c9f174ece None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:28.717346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.718501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.718799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.719248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.720227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:28.720464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.721883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.735755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 279/574] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:28.736675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.770322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:28.775557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 296/575] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:28.776401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8cfecd-44f0-4f64-a182-1f3c9f174ece None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.789909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:28.860065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4cd89-5e19-4a69-ab95-468551fa7261 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.895710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8cfecd-44f0-4f64-a182-1f3c9f174ece None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:28.898242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.898964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.899255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.899540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.900401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:28.900719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.902177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.923322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 280/576] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:28.924531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:28.942402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:28.943110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:28.943648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:28.943971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:28.945294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:28.945592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:28.947078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:28.967397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 297/577] 10.222.0.72 () {56 vars in 1137 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:28.968487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.035022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.035405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.049802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56ba8c2-bbfe-4982-aa39-c7470b97a672 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.078711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a375f71-da33-49ca-a214-e039d8ee1eac None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.087420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.088230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.088514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.088790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.099586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.101121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:29.121320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:29.122328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:29.123272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:29.123272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:29.123272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.123272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:29.125244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.126066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.126365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.126681 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.127342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 281/578] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:29.128218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.141498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:29.150557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:29.151549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:29.152381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 298/579] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:29.153101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.244420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d252873-36b7-424f-9ffb-164b9a976ce7 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.263331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dacbda1-23c6-43cd-9413-1f445a19d780 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.283867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.284664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.284944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.285217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.295167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.298742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.300757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.305043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.322790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:29.323517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:29.324371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:29.325590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:29.325590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:29.325590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.325590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:29.325823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.325823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.325974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 282/580] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:29.326540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.332302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.350788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.353516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:29.376891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:29.377862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:29.378778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:29.378778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:29.378778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.378778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:29.378778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 299/581] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:29.379070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.439748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba8b42c-a6cb-49f0-b2b6-934d0b07ab36 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.483245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.483921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.484212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.484498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.485232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:29.485548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.486978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.489425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55a11f5-eefd-4cb8-8879-fc00c3d64800 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.504421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 283/582] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:29.504421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.530208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.530933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.531211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.531508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.532259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:29.532569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.533974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.548243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 300/583] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:29.548243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.574316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.598892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.627005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ca98b9-b7b2-4292-825d-20c7ad2724e3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.671275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.672224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.672685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.673062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.673804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:29.674117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.675533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.675971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e1ca23-563b-4073-afb0-9c67b414aa81 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.690328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 284/584] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:29.691115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.718445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.719217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.719573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.719922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.720740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:29.721122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.722612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.739300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 301/585] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:29.739300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.804538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3f3f2a-847b-4bab-9d2a-50232b3bce87 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.818549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.843724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.844625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.844917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.845186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.846932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:29.851852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9ccdec-31d5-4125-9aa3-367f9f3f1ce5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:29.853202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.855771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:29.873044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:29.874082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 285/586] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:29.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.893718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:29.894550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:29.894849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:29.895109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:29.907135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:29.908580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:29.926964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:29.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:29.929396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:29.929396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:29.929396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:29.929396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:29.929396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 302/587] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:29.930263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:29.995492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710fbc1f-813e-4082-a207-eaddbce76f82 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.042139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ba857-45ca-48c9-8b65-fd5bdd355898 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.042369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.043738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.044397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.044882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.057645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.060724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.076830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.077625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.077906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.078202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.085417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.086891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:30.088696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:30.089569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:30.090583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:30.091442 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 286/588] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:30.092371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:30.107989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:30.109723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:30.111537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:30.114133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 303/589] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:30.115117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.130065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.138153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.215456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7c1b81-c7e4-4a98-8a23-11ffe1ce0d06 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.230889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a93c37-57d3-48b3-821b-06974dcb8260 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.264577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.265177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.265466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.265748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.266981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:30.267116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.267211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.267576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.267860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.268137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.268382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.268861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:30.269168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.270557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.290172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 304/590] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:30.290172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.290468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 287/591] 10.222.0.72 () {56 vars in 1137 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:30.290468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.364316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.398823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.408423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f084c64-0325-43de-b0bd-14f47ee496d3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.413197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fc75f4-ae69-43d8-ac8b-0b43243aae32 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.443526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.444248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.444536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.444817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.445620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:30.445936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.447446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.447784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.448450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.448735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.449015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.450066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:30.450384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.451830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.468571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 288/592] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:30.468571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.468838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 305/593] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:30.468838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.570118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab953ced-0005-45a4-a9df-8b3bc1f5630c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.589893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc950a1-5c28-49f4-a8f5-b87dbb0e3980 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.605510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2605dc0-330f-4681-94e2-3c771c4d014a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.606310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2605dc0-330f-4681-94e2-3c771c4d014a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.606629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2605dc0-330f-4681-94e2-3c771c4d014a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.606905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2605dc0-330f-4681-94e2-3c771c4d014a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.614553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2605dc0-330f-4681-94e2-3c771c4d014a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.616039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2605dc0-330f-4681-94e2-3c771c4d014a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.618638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.624837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.625638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.625917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.626189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-e2605dc0-330f-4681-94e2-3c771c4d014a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:30.630590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:30.631775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 289/594] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/users/neutron => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dac79f-08ad-4274-8964-98ff600c590d None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dac79f-08ad-4274-8964-98ff600c590d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.638245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dac79f-08ad-4274-8964-98ff600c590d None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.638245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 290/595] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:30.638245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dac79f-08ad-4274-8964-98ff600c590d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.640347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.641903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.650319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:30.663347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:30.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:30.665353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:30.665353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:30.665353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:30.665353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:30.665353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 306/596] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:30.665641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.749300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dac79f-08ad-4274-8964-98ff600c590d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.771422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c41ec8-236e-4c64-a90a-817a1939cfb9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:30.791586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.792477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.792758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.793029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/fb6c2e2ddb3b4d488fa25fa8e258fec0/roles/b032af1c9f7a447a98b75130f5bab37a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.807834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f64ba5-841a-4642-81aa-31e362a465be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:30.808492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=fb6c2e2ddb3b4d488fa25fa8e258fec0, role_id=b032af1c9f7a447a98b75130f5bab37a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.808730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f64ba5-841a-4642-81aa-31e362a465be None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.809021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f64ba5-841a-4642-81aa-31e362a465be None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.809303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f64ba5-841a-4642-81aa-31e362a465be None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/2165eacee8954dce975ba81ba8ac4232/roles/b032af1c9f7a447a98b75130f5bab37a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.809959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.823982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 291/597] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:30 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/fb6c2e2ddb3b4d488fa25fa8e258fec0/roles/b032af1c9f7a447a98b75130f5bab37a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:30.825077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.826665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f64ba5-841a-4642-81aa-31e362a465be None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=2165eacee8954dce975ba81ba8ac4232, role_id=b032af1c9f7a447a98b75130f5bab37a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:30.828219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f64ba5-841a-4642-81aa-31e362a465be None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:30.844770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 307/598] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:30 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/2165eacee8954dce975ba81ba8ac4232/roles/b032af1c9f7a447a98b75130f5bab37a => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:30.846119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2333be74-30a1-4fc7-be8c-66c50aa70ab7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:30.846412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2333be74-30a1-4fc7-be8c-66c50aa70ab7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:30.846726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2333be74-30a1-4fc7-be8c-66c50aa70ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:30.866685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2333be74-30a1-4fc7-be8c-66c50aa70ab7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:30.868078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2333be74-30a1-4fc7-be8c-66c50aa70ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:30.905181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.933415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:30.975832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 308/599] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:30.977068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2333be74-30a1-4fc7-be8c-66c50aa70ab7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:30.988900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbf6dd8-da9f-4146-85f9-627c038eeddc None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.033623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.034317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.034631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.034911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:31.036220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.038351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.066986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 292/600] 10.222.0.72 () {56 vars in 1139 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:31.068825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.107039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2333be74-30a1-4fc7-be8c-66c50aa70ab7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.148722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.149381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.149667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.149946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.150718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:31.151028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.152442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.166583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 309/601] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:31.171222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.198757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:31.217919 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08e8af5-83f3-4764-a70a-0962727b3341 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.242306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:31.265781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.267806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.268325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.268810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.285254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:31.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:31.294377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:31.295229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 293/602] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:31.295904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.296966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3beca8-5045-4834-ade2-7c79fb82495c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.347668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.348332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.348755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.366338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.368914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:31.393197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:31.394475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:31.395596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:31.395596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:31.395596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:31.395596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:31.398856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 310/603] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:31.400380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.418098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13d3c58-9d8b-4f64-90e3-36e7f627b39d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.459921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.461226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.462059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.462516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.472686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.474653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:31.496018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:31.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:31.498256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:31.499644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:31.506838 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 294/604] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:31.507503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.535661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d6874f-ef1d-4ddd-9cd8-818e8fe5113a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.567125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:31.575453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.576139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.576427 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.576709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.577550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:31.577781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.579327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.589479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 311/605] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:31.590370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.651087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91749c80-ba78-41b3-9784-d054bce07533 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.703705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.705029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.705534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.706033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.707029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:31.707554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.709219 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.719006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c3b293-ebf7-4e6e-a483-42e6acd70584 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.722940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 295/606] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:31.724103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.755761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.756959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.757372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.757582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.758307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:31.758665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.760118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.762865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:31.778721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 312/607] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:31.779658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.834515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:31.840400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d10122-8053-45fa-8acb-b4465e833292 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.877965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.879193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.879694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.880174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.890526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff4d759-4415-4482-917e-ad0d72530d85 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:31.893148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.894903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:31.913925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:31.915196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:31.916076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:31.916076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:31.916076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:31.916076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:31.918188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 296/608] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:31.919220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:31.928168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:31.928998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:31.929288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:31.929573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:31.937272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:31.938793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:31.956452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:31.957422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:31.958293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 313/609] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:31.959304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:32.021975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:32.040834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c9edf-8330-47c4-a78c-aabf0a3217a2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:32.070089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0415d0d5-8443-4161-83b8-a495b6dbad0d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:32.079378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:32.080751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:32.081258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:32.081835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/4f1b0819d52d481898b354a5ae182746/roles/b032af1c9f7a447a98b75130f5bab37a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:32.100879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=4f1b0819d52d481898b354a5ae182746, role_id=b032af1c9f7a447a98b75130f5bab37a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:32.102880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:32.109194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:32.109872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:32.110167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:32.111547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:32.111547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:32.111547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:32.112884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:32.117557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:32.123604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 297/610] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:31 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/4f1b0819d52d481898b354a5ae182746/roles/b032af1c9f7a447a98b75130f5bab37a => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:32.124766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:32.133464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 314/611] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:32.139063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:32.252246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be71fbac-a723-4472-9e39-e2cfa5dd6741 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:32.294324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:32.295627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:32.296134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:32.296631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:32.297669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:32.298314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:32.300355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:32.302281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a287c30-9a94-4baf-987f-0d0817a05526 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:32.316508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 298/612] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:32.322614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:32.325620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:32.350057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:32.351031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:32.351190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:32.351509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:32.359973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:32.361504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:32.381349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:32.382746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:32.383696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 315/613] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:32.390245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:32.451642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:32.499127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0351e6d7-6439-4ecf-b0e3-2b178797c9f1 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:32.503167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d101492-03aa-45af-b648-8f7aee74b11e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:32.540956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:32.542599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:32.542897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:32.543267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/ac5254fb5eff4108be9ac3992e05a99c/roles/b032af1c9f7a447a98b75130f5bab37a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:32.557242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:32.557946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:32.558236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:32.558543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:32.559403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:32.559737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:32.561201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:32.573974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=ac5254fb5eff4108be9ac3992e05a99c, role_id=b032af1c9f7a447a98b75130f5bab37a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:32.575768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:32.594262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 316/614] 10.222.0.72 () {56 vars in 1135 bytes} [Mon Jul 11 14:26:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:32.602557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:32.607516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 299/615] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:32 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/ac5254fb5eff4108be9ac3992e05a99c/roles/b032af1c9f7a447a98b75130f5bab37a => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:32.665782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:32.782583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ba9a0-2520-43be-b4c2-729b7ac6367b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:32.823698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:32.824525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:32.825070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:32.825351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:32.839532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:32.841544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:32.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:32.861885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:32.862836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:32.862836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:32.862836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:32.862836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:32.866608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 317/616] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:32.868643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:32.989704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78be7ea-1e4d-433e-a48a-53fb4d761b53 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:33.001835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:33.035689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527534c2-1842-4518-83b7-179f026e91b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:33.036279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527534c2-1842-4518-83b7-179f026e91b3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.036566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527534c2-1842-4518-83b7-179f026e91b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.036857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527534c2-1842-4518-83b7-179f026e91b3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.037658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527534c2-1842-4518-83b7-179f026e91b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:33.037980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527534c2-1842-4518-83b7-179f026e91b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:33.039515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527534c2-1842-4518-83b7-179f026e91b3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:33.059401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 300/617] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:33.065239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.175805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c705c-7ac8-4469-b06b-93b72c80a12e None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.176197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c705c-7ac8-4469-b06b-93b72c80a12e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.176734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c705c-7ac8-4469-b06b-93b72c80a12e None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.178749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 301/618] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:33.186620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28491da-2bd3-4f1a-a82a-6ec3792f622f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.187289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28491da-2bd3-4f1a-a82a-6ec3792f622f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.187587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28491da-2bd3-4f1a-a82a-6ec3792f622f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.210487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:33.222333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e28491da-2bd3-4f1a-a82a-6ec3792f622f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:33.223637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e28491da-2bd3-4f1a-a82a-6ec3792f622f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:33.240698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab195548-b832-4897-b3a3-f56bb9f429e0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:33.296107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704630ef-b883-4bc7-afec-0e42c88d391a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:33.297323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704630ef-b883-4bc7-afec-0e42c88d391a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.297614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704630ef-b883-4bc7-afec-0e42c88d391a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.297896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704630ef-b883-4bc7-afec-0e42c88d391a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.308498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704630ef-b883-4bc7-afec-0e42c88d391a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:33.310152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704630ef-b883-4bc7-afec-0e42c88d391a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-704630ef-b883-4bc7-afec-0e42c88d391a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:33.329229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:33.330370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:33.331292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 318/619] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:33.332483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674b5da1-8053-4a0b-a997-be0a1135c9e5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.332592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674b5da1-8053-4a0b-a997-be0a1135c9e5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.332816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674b5da1-8053-4a0b-a997-be0a1135c9e5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.334075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 302/620] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:33.338229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 319/621] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:33.339471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28491da-2bd3-4f1a-a82a-6ec3792f622f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.340272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5584af0b-82da-48cc-8a7e-8712af854437 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.340782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5584af0b-82da-48cc-8a7e-8712af854437 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.341282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5584af0b-82da-48cc-8a7e-8712af854437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.361628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5584af0b-82da-48cc-8a7e-8712af854437 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:33.363290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5584af0b-82da-48cc-8a7e-8712af854437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:33.450479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28491da-2bd3-4f1a-a82a-6ec3792f622f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:33.450882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 320/622] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:33.452055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5584af0b-82da-48cc-8a7e-8712af854437 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.453867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:33.483697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:33.484317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.484939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.485223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.486024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:33.486519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:33.487965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:33.500738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 303/623] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:33.504157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.561687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:33.590878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5584af0b-82da-48cc-8a7e-8712af854437 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:33.611766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269cf33-cd30-4409-8ff4-630ff0e52763 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:33.631197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:33.632810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.633423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.633901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.648217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:33.653915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:33.657899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.657899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.657899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:33.660953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:33.661976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:33.663407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:33.666495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 321/624] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:33.673583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.682667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:33.698029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:33.699079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:33.700196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:33.706763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 304/625] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:33.708340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.749417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:33.791253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0ff75-8b8a-4e4e-b43d-be3e194dc28e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:33.829791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:33.831142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.831626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.832095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/055512e4e3744833ac6a9e61d22f0b63/roles/b032af1c9f7a447a98b75130f5bab37a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.836396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:33.850308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=055512e4e3744833ac6a9e61d22f0b63, role_id=b032af1c9f7a447a98b75130f5bab37a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:33.852268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:33.869807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 322/626] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:33 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/055512e4e3744833ac6a9e61d22f0b63/roles/b032af1c9f7a447a98b75130f5bab37a => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:33.871208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.889619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0003d0-aaf7-4c2d-9725-24e245147d32 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:33.930380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:33.931094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:33.931382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:33.931659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:33.932442 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:33.932754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:33.934187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:33.944689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 305/627] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:33.950186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:33.996447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b07e3-fa71-47d2-9d8b-31776fd71bd6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.036861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.038581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.039097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.039589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.040685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:34.041237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.043074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.050849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:34.053235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 323/628] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:34.060292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:34.079233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0eb4b-67ff-4f77-9158-5af1c89876f3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.120010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.121073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.121532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.121961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.135417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.137251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:34.156687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:34.157852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:34.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:34.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:34.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:34.158992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:34.161172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 306/629] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:34.167595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:34.174847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:34.272169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e072224-da11-4a6f-bba6-35ed1c07867f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.311809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cc4de1-60c0-4982-87f7-a55e3d1dabe8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.338213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.339338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.339780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.340210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.352456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.354184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:34.373677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:34.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:34.375948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:34.375948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:34.375948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:34.375948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:34.377944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 324/630] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:34.383727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:34.389046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.389732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.390021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.390301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.391073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:34.391490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.392943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.398754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:34.424949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 307/631] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:34.447520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:34.498812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:34.506789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9673b869-9503-4c66-a1b0-be23273d0d46 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.580445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.581135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.581416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.581690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.582423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:34.582752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.584156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.617500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 325/632] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:34.623439 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:34.630957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc392bc-31ca-4dd5-83ab-84f36d906a06 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.684274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.684274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.684274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.684274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] PATH_INFO: `/v3/users/glance` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.700477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.704408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:34.724097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:34.725807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 11 14:26:34.726800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:34.730101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 308/633] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:34 2022] GET /identity/v3/users/glance => generated 84 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:34.735976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:34.811753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:34.845345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca2bb4-a21c-48b3-a2d8-fc80ac8b7353 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.872864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdae6ce-906d-4540-a2d0-20418157694b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:34.886919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02840e38-de5c-4ae6-868d-71c578ce12cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.888072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02840e38-de5c-4ae6-868d-71c578ce12cd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.888513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02840e38-de5c-4ae6-868d-71c578ce12cd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.889180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02840e38-de5c-4ae6-868d-71c578ce12cd None admin] PATH_INFO: `/v3/users/nova` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.899922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02840e38-de5c-4ae6-868d-71c578ce12cd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.902701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02840e38-de5c-4ae6-868d-71c578ce12cd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.913591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b2710-0677-4c27-832d-b2af6c0ff646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:34.914510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b2710-0677-4c27-832d-b2af6c0ff646 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.914904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b2710-0677-4c27-832d-b2af6c0ff646 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.915288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b2710-0677-4c27-832d-b2af6c0ff646 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.916271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b2710-0677-4c27-832d-b2af6c0ff646 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:34.916720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b2710-0677-4c27-832d-b2af6c0ff646 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:34.918273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9b2710-0677-4c27-832d-b2af6c0ff646 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-02840e38-de5c-4ae6-868d-71c578ce12cd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:34.924312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:34.925455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:34.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:34.929392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 326/634] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:26:34 2022] GET /identity/v3/users/nova => generated 82 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:34.930976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bd92a-c336-4cf6-951d-74c4059581d8 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.931402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bd92a-c336-4cf6-951d-74c4059581d8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.931827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bd92a-c336-4cf6-951d-74c4059581d8 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.933521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 327/635] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:34.934402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5bd92a-c336-4cf6-951d-74c4059581d8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:34.954488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 309/636] 10.222.0.72 () {56 vars in 1137 bytes} [Mon Jul 11 14:26:34 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:34.954488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4514e16-699e-4944-917f-0b5d2b1158a2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:34.954488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4514e16-699e-4944-917f-0b5d2b1158a2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:34.954488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4514e16-699e-4944-917f-0b5d2b1158a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:34.969857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:34.971467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c4514e16-699e-4944-917f-0b5d2b1158a2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:34.972777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c4514e16-699e-4944-917f-0b5d2b1158a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:35.049349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 310/637] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:35.055344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4514e16-699e-4944-917f-0b5d2b1158a2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:35.080120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5bd92a-c336-4cf6-951d-74c4059581d8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:35.125001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542681ba-e421-4004-849f-04136bec31fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:35.125823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542681ba-e421-4004-849f-04136bec31fa None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.126139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542681ba-e421-4004-849f-04136bec31fa None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.126464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542681ba-e421-4004-849f-04136bec31fa None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.127406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542681ba-e421-4004-849f-04136bec31fa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:35.127737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542681ba-e421-4004-849f-04136bec31fa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:35.129195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542681ba-e421-4004-849f-04136bec31fa None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:35.139580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:35.153242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 328/638] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:26:34 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:35.158788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542681ba-e421-4004-849f-04136bec31fa None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:35.241953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:35.249569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4514e16-699e-4944-917f-0b5d2b1158a2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:35.270780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542681ba-e421-4004-849f-04136bec31fa None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:35.288142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02266255-6170-4e55-b1ee-2656c09cae5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:35.288827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02266255-6170-4e55-b1ee-2656c09cae5e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.289117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02266255-6170-4e55-b1ee-2656c09cae5e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.289401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02266255-6170-4e55-b1ee-2656c09cae5e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.316039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:35.317128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.317415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.317832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.335382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:35.346614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:35.347651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 329/639] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:35.353977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:35.366960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02266255-6170-4e55-b1ee-2656c09cae5e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:35.368713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02266255-6170-4e55-b1ee-2656c09cae5e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:35.419457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:35.467064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84877cf-a1c4-463c-8253-89edbeed623e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:35.483457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 311/640] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:35.485156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02100081-e51e-46bf-9c4f-531a215c4510 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.485532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02100081-e51e-46bf-9c4f-531a215c4510 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.485886 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02100081-e51e-46bf-9c4f-531a215c4510 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.487443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 312/641] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:35.488153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02100081-e51e-46bf-9c4f-531a215c4510 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:35.511766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eadade-f93c-4d9f-8a71-450a98c598cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:35.512668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eadade-f93c-4d9f-8a71-450a98c598cc None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.512954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eadade-f93c-4d9f-8a71-450a98c598cc None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.513235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eadade-f93c-4d9f-8a71-450a98c598cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.523173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:35.590621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eadade-f93c-4d9f-8a71-450a98c598cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:35.592013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eadade-f93c-4d9f-8a71-450a98c598cc None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:35.601649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02100081-e51e-46bf-9c4f-531a215c4510 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:35.642785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:35.643539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.643788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.644070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.644829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:35.645145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:35.646561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:35.663443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 313/642] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:35.663443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:35.666748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:35.717853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 330/643] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:35.717853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be79d5c9-e69d-461b-9a7f-ff52d8612eb2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.717853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be79d5c9-e69d-461b-9a7f-ff52d8612eb2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.717853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be79d5c9-e69d-461b-9a7f-ff52d8612eb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.733286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-be79d5c9-e69d-461b-9a7f-ff52d8612eb2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:35.735223 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-be79d5c9-e69d-461b-9a7f-ff52d8612eb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:35.766671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:35.786385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53736350-4b42-4c2c-9c56-7cd1c622f157 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:35.808691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 331/644] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:35.809652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be79d5c9-e69d-461b-9a7f-ff52d8612eb2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:35.834344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:35.835746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.836034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.836313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.847739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:35.849256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:35.867931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:35.868991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:35.869899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:35.869899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:35.869899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:35.869899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:35.869899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 314/645] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:35.870188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:35.936408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be79d5c9-e69d-461b-9a7f-ff52d8612eb2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:35.937747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:35.976073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:35.976896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:35.977187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:35.977467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:35.989954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:35.991955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:35.993761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d031f005-7b4e-47e3-b1ea-e384d4d37046 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:36.011931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:36.013656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:36.016864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:36.016864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:36.016864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:36.016864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:36.016864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 332/646] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:36.016864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.038353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.038928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.040156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.040372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.040594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.052739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.054768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:36.073786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:36.075055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:36.076007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:36.076007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:36.076007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:36.076007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:36.076007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 315/647] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:36.076912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.128790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bff6a-0439-4c9e-87c7-e45b7c9501fb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.164534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.165469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.165760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.166045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.178249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:36.186112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:36.187186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:36.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 333/648] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:36.189169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.199351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db93ccd9-437f-4fa5-9270-c6ab1b203c2d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.210481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.237982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.238683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.238969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.239249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.239991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:36.240298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.241781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.253913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 316/649] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:36.254779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.285801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.305467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1b834d-79b3-404b-bbb2-f276581ee81f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.346834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.347511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.347801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.348092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.348826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:36.349135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.350628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.359647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043fbf8-b661-491f-a41c-34d7072f9ea2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.363770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 334/650] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:36.364603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.400435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0455ff70-0884-4777-a160-3acfc2eeda3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.401118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455ff70-0884-4777-a160-3acfc2eeda3b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.401404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455ff70-0884-4777-a160-3acfc2eeda3b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.401687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455ff70-0884-4777-a160-3acfc2eeda3b None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.402423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0455ff70-0884-4777-a160-3acfc2eeda3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:36.402769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0455ff70-0884-4777-a160-3acfc2eeda3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.404164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0455ff70-0884-4777-a160-3acfc2eeda3b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.415848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 317/651] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:36.417133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243abc85-d76b-46b8-a039-81ae72171d11 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.417432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243abc85-d76b-46b8-a039-81ae72171d11 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.417717 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243abc85-d76b-46b8-a039-81ae72171d11 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.422687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 318/652] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:36.423453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243abc85-d76b-46b8-a039-81ae72171d11 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.455006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.466656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9437b4d-cfd8-44cc-9090-edc053d0d102 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.500426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.501101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.501387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.501670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.502420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:36.502761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.504157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.512987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 335/653] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:36.513826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.516437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.536138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243abc85-d76b-46b8-a039-81ae72171d11 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.571509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.572304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.572660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.572974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.580452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.581900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:36.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:36.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:36.598530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 319/654] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:36.599024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.623661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7477426-910c-4c44-890d-5a673d8c0c46 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.659170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16df06f6-210d-4fc8-acb1-4feec3173a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.660243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16df06f6-210d-4fc8-acb1-4feec3173a3f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.660704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16df06f6-210d-4fc8-acb1-4feec3173a3f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.660983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16df06f6-210d-4fc8-acb1-4feec3173a3f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.668678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16df06f6-210d-4fc8-acb1-4feec3173a3f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.670142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16df06f6-210d-4fc8-acb1-4feec3173a3f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-16df06f6-210d-4fc8-acb1-4feec3173a3f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:36.687780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:36.688799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:36.689740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 336/655] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:36.690341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d176f9-f165-4183-a7d3-e1ddcdabb6fa None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.690882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d176f9-f165-4183-a7d3-e1ddcdabb6fa None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.690882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d176f9-f165-4183-a7d3-e1ddcdabb6fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.695326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.708896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-94d176f9-f165-4183-a7d3-e1ddcdabb6fa None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:36.709221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2101a1a-7425-4e60-9726-494b4b441370 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.710039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-94d176f9-f165-4183-a7d3-e1ddcdabb6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:36.743829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.744629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.744916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.745191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.752017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.753476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.767015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:36.767986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:36.768911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 320/656] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:36.770108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.784491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 337/657] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:36.785361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d176f9-f165-4183-a7d3-e1ddcdabb6fa None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.874635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38bdb9-5665-452e-8c42-01342e13490e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.898066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d176f9-f165-4183-a7d3-e1ddcdabb6fa None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:36.908786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.909668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.909948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.910220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.921263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.922875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.936716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:36.937393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:36.937701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:36.938007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:36.938873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:36.939188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:36.940628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:36.944576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:36.948390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:36.949360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:36.950945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:36.950945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:36.950945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:36.950945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:36.950945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 321/658] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:36.951241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:36.955043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 338/659] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:36.955977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.013576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:37.054779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc4b7c-b76f-4a62-88d6-2af25d7a51da None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.062117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b18303-fef2-4364-b2ef-436cdd29a2e9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.090641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.091314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.091596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.091872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.092712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:37.093074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.094462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.095930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.096589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.096869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.097146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.097933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:37.098244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.099711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.113047 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 339/660] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:37.113879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.114577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 322/661] 10.222.0.72 () {56 vars in 1137 bytes} [Mon Jul 11 14:26:36 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:37.116193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.176366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:37.220008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067db1f8-57cc-469c-b020-2a6c7db8a058 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.221763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ac082-d4b6-4c5d-92d8-121707c701da None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.235932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:37.255697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.256420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.256704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.256978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.257700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:37.258008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.258557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.259353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.259466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.259635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.259909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.271054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 323/662] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:37.271445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:37.272355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:37.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:37.281662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 340/663] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:37.282653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.400129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aed302-605e-4510-8870-c43a26c0afce None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.403676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f84a2b-71f8-4289-adc9-8d5d104ef86a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.422299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:37.435516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.436180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.436462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.436735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.437567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=2165eacee8954dce975ba81ba8ac4232) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:37.437876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.439433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.441879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.442579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.442867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.443156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.444044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=fb6c2e2ddb3b4d488fa25fa8e258fec0) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:37.444363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.445412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 341/664] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/role_assignments?user.id=2165eacee8954dce975ba81ba8ac4232&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:37.445806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.446309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.452607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 324/665] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/role_assignments?user.id=fb6c2e2ddb3b4d488fa25fa8e258fec0&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:37.453490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.522655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:37.596486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a578c-6036-424a-b63c-06a1b66dcb86 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.612468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4213b7a8-b0a5-4e5e-8597-5f3629f47c15 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.648315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.648315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.648613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.648859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.658766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.660037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.660548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.661024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.671818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.673846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:37.690779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:37.691792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:37.692748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 325/666] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/users/neutron => generated 85 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:37.693359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.749535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.751066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.762818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:37.790960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:37.822559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1737df-abf2-406e-a90c-337137a72c12 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:37.858575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:37.859259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:37.859607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:37.859900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:37.860325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 342/667] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:37.861075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:37.861184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.861382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:37.867248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:37.877290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 326/668] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:37.883505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:37.968092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e4c055-f089-4b2d-9d5b-2705783712c8 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.002380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec86ca-943d-4b61-8abc-18dcd76e5da6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.006925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.011217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.013121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.013620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.014097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.015237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:38.015850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.017551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.038723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 343/669] 10.222.0.72 () {56 vars in 1139 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:38.040219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.045266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.047313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.048131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.048576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.048864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.065171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.066694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:38.089907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:38.091115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:38.092008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:38.092008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:38.092008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:38.092008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:38.094283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 327/670] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:38.095396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.183714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5452434-bde4-4a98-806e-a3d6f9b6fb78 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.221057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.222352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.222865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.223336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.224341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dcf523-53bd-44a0-a820-f1a3ff066677 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.236082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.238098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:38.258167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:38.259327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:38.260608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:38.260608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:38.260608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:38.260608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:38.266451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.267376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.267555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 344/671] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:38.268024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.268438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.268859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.304309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.327319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.364972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.367464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.397560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f63057-c651-42ec-bfaa-4b2dda5c4402 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.435939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.436980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.437476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.438031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.439088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:38.439626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.441215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.455759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 345/672] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:38.457087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.493236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 328/673] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:38.494411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.555208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.572995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3829e7b5-a865-421b-b42f-f1823d719f78 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.581558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.609640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5b587-3414-44c7-a6d5-0399985c870a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.612238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.613585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.613585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.613585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.614599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:38.615015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.616591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.634278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 346/674] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:38.635433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.649890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.651090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.651557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.652009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.659984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.661787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:38.680501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:38.681527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:38.682634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:38.684536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 329/675] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:38.685483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.782158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76141850-92a3-4377-a829-a3ca245d6b88 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.814989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdfbeb3-129f-4be5-9fb2-025c1441a54e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:38.823549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.824598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.825889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.826317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.837663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.839887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.841738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.860752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:38.861766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:38.862162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:38.862495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:38.863492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:38.864585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:38.864585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:38.864585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:38.864585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:38.864844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:38.867166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 347/676] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:38.868218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:38.876246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:38.877889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:38.892720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:38.900255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:38.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:38.902592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:38.906577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 330/677] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:38.907560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.005953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcad8290-db4d-49d0-bf30-ae185c5717a3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.031214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2994afb-4470-4f87-a295-fcb89630db2e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.045497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.046386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.046846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.047282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.048214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:39.048664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.050262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.064156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 348/678] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:39.065213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.080026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.081388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.082017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.082466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.083512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:39.083969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.086887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.103566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 331/679] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:39.104907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.150414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:39.168450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:39.201518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c27290-d2d7-47bb-9ee8-44376ebac2ce None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.236216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965af1da-d80d-411e-874c-8b8cf2307ef9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.251450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.252149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.252437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.252854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.253872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:39.254192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.256544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.281518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.282568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.283054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.283562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.284745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=4f1b0819d52d481898b354a5ae182746) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:39.285014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 349/680] 10.222.0.72 () {56 vars in 1135 bytes} [Mon Jul 11 14:26:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:39.285515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.285945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.288084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.296812 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 332/681] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:39 2022] GET /identity/v3/role_assignments?user.id=4f1b0819d52d481898b354a5ae182746&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:39.298130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.512948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:39.513447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3122c6-0844-421d-ac94-9436c88bd0f2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.516487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66bd1a-a800-4744-aad8-8f1be0483085 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.558752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.560081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.560554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.561001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a500bf27-7758-4cae-96c4-f73bab1a72bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.561222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.561784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a500bf27-7758-4cae-96c4-f73bab1a72bf None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.562157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a500bf27-7758-4cae-96c4-f73bab1a72bf None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.562487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a500bf27-7758-4cae-96c4-f73bab1a72bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.565777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:39.574497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.576635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:39.593735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:39.594773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:39.595656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:39.602708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 350/682] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:39.605028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.652295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a500bf27-7758-4cae-96c4-f73bab1a72bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.653821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a500bf27-7758-4cae-96c4-f73bab1a72bf None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.732406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fc549c-bf22-4437-8e95-827f0bbd9bbb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.769680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.770260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.770581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.770869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.772208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:39.772525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.774023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.774155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:39.782647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 333/683] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 485 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:39.784317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a87a394-b8ea-45cc-be96-5dac59113185 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.784607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a87a394-b8ea-45cc-be96-5dac59113185 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.784884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a87a394-b8ea-45cc-be96-5dac59113185 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.790336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 334/684] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:39.790336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e01b7-b8d1-4f90-bff7-7a344630a072 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.790336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e01b7-b8d1-4f90-bff7-7a344630a072 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.790715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 351/685] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:39.790808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e01b7-b8d1-4f90-bff7-7a344630a072 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.790808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 335/686] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:39 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:39.794005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.794400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b1a68-364b-49f3-87a7-971a7b8efedf None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.794725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b1a68-364b-49f3-87a7-971a7b8efedf None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.795006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b1a68-364b-49f3-87a7-971a7b8efedf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.814015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fc0b1a68-364b-49f3-87a7-971a7b8efedf None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:39.815484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fc0b1a68-364b-49f3-87a7-971a7b8efedf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:39.857726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:39.898221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 336/687] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:39.899130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b1a68-364b-49f3-87a7-971a7b8efedf None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:39.925462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b4c92e-a6e8-4872-82ec-015468751be2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:39.962673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f6c1d0-24b3-49f4-a0d1-dba684bb4129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:39.963574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f6c1d0-24b3-49f4-a0d1-dba684bb4129 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.963939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f6c1d0-24b3-49f4-a0d1-dba684bb4129 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.964290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f6c1d0-24b3-49f4-a0d1-dba684bb4129 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:39.975024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f6c1d0-24b3-49f4-a0d1-dba684bb4129 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:39.976598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f6c1d0-24b3-49f4-a0d1-dba684bb4129 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-16f6c1d0-24b3-49f4-a0d1-dba684bb4129 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:39.994789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:39.995950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:39.996859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:39.996859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:39.996859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:39.996859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:39.996859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 352/688] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:39.997672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5af9e7e-d0c5-431d-ad03-c65e749ad30c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:39.997985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5af9e7e-d0c5-431d-ad03-c65e749ad30c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:39.998270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5af9e7e-d0c5-431d-ad03-c65e749ad30c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.019325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d5af9e7e-d0c5-431d-ad03-c65e749ad30c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:40.020579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d5af9e7e-d0c5-431d-ad03-c65e749ad30c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:40.023734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b1a68-364b-49f3-87a7-971a7b8efedf None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.061174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:40.070801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.071629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.071923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.072468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.073590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=ac5254fb5eff4108be9ac3992e05a99c) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:40.073913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.076109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.094480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 337/689] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:39 2022] GET /identity/v3/role_assignments?user.id=ac5254fb5eff4108be9ac3992e05a99c&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:40.094480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.097932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 353/690] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:40.099078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5af9e7e-d0c5-431d-ad03-c65e749ad30c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.155114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:40.236597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5af9e7e-d0c5-431d-ad03-c65e749ad30c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.257883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f54290-f546-43cd-a3ed-3f1cbfa7c4bd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.281476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.281476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.281476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.281476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.282181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:40.282594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.284003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.302227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 354/691] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:40.303316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.309352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.310459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.310944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.311406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.323718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:40.331926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:40.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:40.334121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:40.336466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 338/692] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:40.337501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.415613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:40.426122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:40.432618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ebe30-6bea-4f35-93ad-cce156366143 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.474253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbab082-502d-45cf-8958-a0462df0fd5b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.475314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.476965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.476965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.476965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.495450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.497127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:40.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:40.521031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:40.522116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:40.522116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:40.522116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:40.522116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:40.527614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 355/693] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:40.527614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.532515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.533573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.534028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.534490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.546134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.548384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:40.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:40.571848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:40.572996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:40.575511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 339/694] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:40.576574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.662838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c43d64-e026-413d-a0ef-108a03fe2862 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.697945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe89af27-0d39-4669-b7e6-1b09b7ee8d34 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.710404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.711968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.712410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.712837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.713940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:40.714407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.714927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:40.716698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.728882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 356/695] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:40.730109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.739939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.741551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.741551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.741551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.742916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:40.742916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.744779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.751858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:40.764915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 340/696] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:40.764915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.875585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c7aee4-5baf-4af3-8294-61b55e0cc7dd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.877521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c94cc-2b6c-4fa2-8357-46d0c1a1f997 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:40.911511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.913040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.913475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.913881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.915594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:40.916569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:40.917143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:40.917606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:40.918628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:40.919126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.920755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.926862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:40.928930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:40.935208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 357/697] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:40.936462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:40.947832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:40.948770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:40.950549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:40.950549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:40.950549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:40.950549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:40.950549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 341/698] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:40.950847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:40.980701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:41.020420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:41.056249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b3f64-063f-46f4-b0e4-3842dd2d264d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:41.059561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fcf552-6880-4004-a5f7-564e3f979454 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:41.094763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:41.095496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.095791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:41.095960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.096363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:41.097015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.097335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=055512e4e3744833ac6a9e61d22f0b63) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:41.097671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.097775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:41.098212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:41.099131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:41.099371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:41.100967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:41.106342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 342/699] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:40 2022] GET /identity/v3/role_assignments?user.id=055512e4e3744833ac6a9e61d22f0b63&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:41.107369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:41.138203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 358/700] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:26:40 2022] POST /identity/v3/users => generated 254 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:41.218267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:41.229284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42658e0-73f9-4ae3-8928-3148124d28d3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:41.270286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:41.271020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.271309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.272127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:41.272898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:41.273215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:41.274667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:41.289583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 343/701] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:41.294920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:41.443512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45301937-2831-4489-8f26-1fc47108e82e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:41.473963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:41.487358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce78cac3-1cc7-470c-ad5e-36657b6439e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:41.488448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce78cac3-1cc7-470c-ad5e-36657b6439e8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.488791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce78cac3-1cc7-470c-ad5e-36657b6439e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.489091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce78cac3-1cc7-470c-ad5e-36657b6439e8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:41.497467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce78cac3-1cc7-470c-ad5e-36657b6439e8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:41.498998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce78cac3-1cc7-470c-ad5e-36657b6439e8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ce78cac3-1cc7-470c-ad5e-36657b6439e8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:41.514238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:41.515308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:41.517487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 359/702] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:26:41 2022] GET /identity/v3/users/nova => generated 82 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:41.521661 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:41.639883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646ec888-fa5e-4815-8321-0b61ce33bb62 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:41.681638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a83abb2-e530-4864-bf60-090c8143ad80 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.682217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a83abb2-e530-4864-bf60-090c8143ad80 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.682761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a83abb2-e530-4864-bf60-090c8143ad80 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:41.684588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 360/703] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:41.688302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:41.689494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.689792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.690079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:41.690994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:41.691314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:41.691629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57a431-25ca-4f8e-8a7d-499d7b37ff26 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.692122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57a431-25ca-4f8e-8a7d-499d7b37ff26 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.692589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57a431-25ca-4f8e-8a7d-499d7b37ff26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:41.692763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:41.722610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1b57a431-25ca-4f8e-8a7d-499d7b37ff26 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:41.726704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1b57a431-25ca-4f8e-8a7d-499d7b37ff26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:41.728227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 344/704] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:26:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:41.735420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:41.754097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:41.839829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 361/705] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:41.861826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b57a431-25ca-4f8e-8a7d-499d7b37ff26 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:41.940012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435732c0-e63e-435d-88c3-d441ba00cf46 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:41.993839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:41.996167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:41.997439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:41.998551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.001365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b57a431-25ca-4f8e-8a7d-499d7b37ff26 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:42.032369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:42.046042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:42.047317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:42.047817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:42.048656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.049569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:42.051553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:42.071016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:42.072097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:42.072974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:42.076369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 362/706] 10.222.0.72 () {56 vars in 1129 bytes} [Mon Jul 11 14:26:41 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:42.084266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:42.176806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:42.180393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:42.210303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:42.310952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59791345-455c-472f-9a79-0c10184a3423 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:42.361365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:42.362088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:42.362368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:42.362861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.363588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:42.363896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:42.365354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:42.376777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 363/707] 10.222.0.72 () {56 vars in 1124 bytes} [Mon Jul 11 14:26:42 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:42.384761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 345/708] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 650 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:42.385757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:42.390088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:42.478085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:42.555946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3c30ee-ec1f-4c7b-b8b2-bdc3fa03bc62 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:42.573343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e42a5-5f97-4f88-a372-c49684134a13 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:42.587227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:42.603772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e42c4f-61f4-4a39-b616-da11dde6875a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:42.604784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e42c4f-61f4-4a39-b616-da11dde6875a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:42.605339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e42c4f-61f4-4a39-b616-da11dde6875a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:42.605811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e42c4f-61f4-4a39-b616-da11dde6875a None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.606787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e42c4f-61f4-4a39-b616-da11dde6875a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:42.607543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e42c4f-61f4-4a39-b616-da11dde6875a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:42.610264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e42c4f-61f4-4a39-b616-da11dde6875a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:42.617705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 364/709] 10.222.0.72 () {56 vars in 1124 bytes} [Mon Jul 11 14:26:42 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:42.619242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914fed6c-9c53-4b2f-bb86-620ce65ab687 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:42.619741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914fed6c-9c53-4b2f-bb86-620ce65ab687 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:42.620211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914fed6c-9c53-4b2f-bb86-620ce65ab687 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.622049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 365/710] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:42.623327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:42.624416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:42.624903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:42.625594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.634645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc275f-5505-4b61-a957-a451a3d96303 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:42.635231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc275f-5505-4b61-a957-a451a3d96303 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:42.635683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc275f-5505-4b61-a957-a451a3d96303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.662556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:42.664391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:42.669262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5fbc275f-5505-4b61-a957-a451a3d96303 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:42.672363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5fbc275f-5505-4b61-a957-a451a3d96303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:42.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:42.703459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:42.704585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:42.704585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:42.704585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:42.704585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:42.715744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 346/711] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:42.722099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:42.798359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 366/712] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 11 14:26:42.840776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc275f-5505-4b61-a957-a451a3d96303 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:42.901460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:42.918484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc0ce3-7d85-4abb-83af-c835230c9221 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:42.975391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:42.976368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:42.977658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:42.978110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:42.979492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:42.981566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:42.983171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:42.998414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 347/713] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:43.007500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:43.016673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc275f-5505-4b61-a957-a451a3d96303 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:43.057459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:43.058466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.058625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:43.058942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.059521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] PATH_INFO: `/v3/services/network` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.060171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:43.061746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:43.080355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:43.081698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:43.082768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:43.095662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 367/714] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:42 2022] GET /identity/v3/services/network => generated 88 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:43.098546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:43.210985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef4057-a74c-4f7f-857f-3db25856d230 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:43.258191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb470471-f960-40d2-b931-4136d6c45054 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:43.281659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:43.297830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:43.298678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.299242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.299656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.301554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] RBAC: Adding query filter params (name=network) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:43.301973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:43.304272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:43.307756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:43.308599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.308893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.309181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.309775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 368/715] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:26:43 2022] GET /identity/v3/services?name=network => generated 124 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:43.315315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:43.332209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:43.333788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:43.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:43.354855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:43.355783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 348/716] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:43.362735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:43.534830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:43.537723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc94ce-e8cc-4436-86f5-99960457b774 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:43.589715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9295b22-bb11-462f-827b-28d6c9671405 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:43.594469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:43.595191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.595481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.595770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.596564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] RBAC: Adding query filter params (type=network) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:43.596879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:43.598360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:43.609369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 369/717] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:26:43 2022] GET /identity/v3/services?type=network => generated 124 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:43.654702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e162990-ccb5-43e5-8f79-c1291c219ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:43.655401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e162990-ccb5-43e5-8f79-c1291c219ed0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.655691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e162990-ccb5-43e5-8f79-c1291c219ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.656005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e162990-ccb5-43e5-8f79-c1291c219ed0 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.656816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e162990-ccb5-43e5-8f79-c1291c219ed0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:43.657137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e162990-ccb5-43e5-8f79-c1291c219ed0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:43.658842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e162990-ccb5-43e5-8f79-c1291c219ed0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:43.678459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 349/718] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:43.680040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:43.743299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:43.891228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8188c5-6d20-4e29-82b1-68641128af67 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.891469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8188c5-6d20-4e29-82b1-68641128af67 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.891851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8188c5-6d20-4e29-82b1-68641128af67 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.894220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8d8b24-a178-4fcc-81b3-c97f52b0c35e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:43.894549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 350/719] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:43.900105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb3089-93fc-4f55-8494-9601835e2db7 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.900105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb3089-93fc-4f55-8494-9601835e2db7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.900105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb3089-93fc-4f55-8494-9601835e2db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.925412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-17bb3089-93fc-4f55-8494-9601835e2db7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:43.926314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-17bb3089-93fc-4f55-8494-9601835e2db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:43.968880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:43.970472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:43.971636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:43.972423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:43.975200 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=2165eacee8954dce975ba81ba8ac4232) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:43.975976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:43.979329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:43.993177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 370/720] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:43 2022] GET /identity/v3/role_assignments?user.id=2165eacee8954dce975ba81ba8ac4232&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:44.028094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 351/721] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:44.079690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:44.109492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff8f0ad-aa2e-448b-acc5-1ebb0cc3bbf1 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:44.109907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff8f0ad-aa2e-448b-acc5-1ebb0cc3bbf1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:44.110051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff8f0ad-aa2e-448b-acc5-1ebb0cc3bbf1 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:44.111902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 352/722] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:44.118113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d008e28-896b-4e59-8760-4ee805110ba6 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:44.118528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d008e28-896b-4e59-8760-4ee805110ba6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:44.118730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d008e28-896b-4e59-8760-4ee805110ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:44.137408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8d008e28-896b-4e59-8760-4ee805110ba6 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:44.138677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8d008e28-896b-4e59-8760-4ee805110ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:44.244981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:44.251461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baec05d-1594-4ea2-91b1-9e647d8ab74b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:44.251722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:44.253533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 353/723] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 11 14:26:44.287060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:44.288066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:44.288419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:44.288767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:44.297558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d008e28-896b-4e59-8760-4ee805110ba6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:44.300212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:44.308607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:44.309587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:44.310442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 371/724] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:44.315829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:44.452842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e4d22b-f87f-420f-b6ab-e7401dcb7355 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:44.492022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d008e28-896b-4e59-8760-4ee805110ba6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:44.499261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:44.499865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:44.500155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:44.500435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:44.501203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:44.501512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:44.502956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:44.513007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 372/725] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:44.523729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:44.536737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:44.538535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:44.539030 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:44.539510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:44.551681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:44.561107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:44.562056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:44.562955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:44.565540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 354/726] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:44.570843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:44.572632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:44.675455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f93155-0d21-4c76-bda1-cfda4802d9ec None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:44.715737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:44.716922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:44.717416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:44.718848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:44.740986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:44.742458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:44.747961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:44.771143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:44.772149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:44.773059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:44.773059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:44.773059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:44.773059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:44.774335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 373/727] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:44.781950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:44.827372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeaaf78-071e-40bd-a794-11cd9eee0ad5 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:44.889009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:44.889687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:44.889972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:44.890252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:44.891034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:44.891815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:44.893208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:44.904912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 355/728] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:44.910069 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:44.923525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:44.996738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166e7fd5-9026-4bab-8368-8e04a84a9bfa None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.068766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:45.069356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.069642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.069923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.070686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:45.070999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:45.072384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:45.093572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 374/729] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:45.099009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:45.117311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35b2d37-1e73-4083-afb5-f72b3b9cd1a9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.159253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:45.184624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:45.185434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.185718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.185996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.205497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:45.207588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:45.228855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:45.231058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:45.233240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:45.233240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:45.233240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:45.233240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:45.236580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 356/730] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:45.238298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:45.293896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25112a3d-1df5-4e84-a166-6eb0dd4102b3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.334922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:45.347964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3fd2b8-d814-42ad-a9bb-1f5e85fa5da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:45.349057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3fd2b8-d814-42ad-a9bb-1f5e85fa5da4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.350021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3fd2b8-d814-42ad-a9bb-1f5e85fa5da4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.350467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3fd2b8-d814-42ad-a9bb-1f5e85fa5da4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.359544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3fd2b8-d814-42ad-a9bb-1f5e85fa5da4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:45.361475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3fd2b8-d814-42ad-a9bb-1f5e85fa5da4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ce3fd2b8-d814-42ad-a9bb-1f5e85fa5da4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:45.378694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:45.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:45.381585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:45.388376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39d4a8-b528-4b99-b158-3f83232ca949 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.388933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 375/731] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:45.390590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88ece2e-bdf6-420b-9060-b69d4b1661a1 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.391035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88ece2e-bdf6-420b-9060-b69d4b1661a1 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.391464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88ece2e-bdf6-420b-9060-b69d4b1661a1 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.393209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 376/732] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:45.394180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88ece2e-bdf6-420b-9060-b69d4b1661a1 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:45.428549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8078a3-ed0e-47bc-a4ca-8cd8473517b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:45.429278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8078a3-ed0e-47bc-a4ca-8cd8473517b3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.429567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8078a3-ed0e-47bc-a4ca-8cd8473517b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.429865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8078a3-ed0e-47bc-a4ca-8cd8473517b3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.430856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8078a3-ed0e-47bc-a4ca-8cd8473517b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:45.431552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8078a3-ed0e-47bc-a4ca-8cd8473517b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:45.433242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8078a3-ed0e-47bc-a4ca-8cd8473517b3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:45.450881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 357/733] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:45.452463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917be50d-7ad3-47c7-9496-d08b1938d43c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.452992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917be50d-7ad3-47c7-9496-d08b1938d43c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.453570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917be50d-7ad3-47c7-9496-d08b1938d43c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.470249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-917be50d-7ad3-47c7-9496-d08b1938d43c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:45.471752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-917be50d-7ad3-47c7-9496-d08b1938d43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:45.500130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:45.523106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88ece2e-bdf6-420b-9060-b69d4b1661a1 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.550784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 358/734] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:45.551980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917be50d-7ad3-47c7-9496-d08b1938d43c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:45.564096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:45.564783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.565064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.565373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.566178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:45.566513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:45.570476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:45.596011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 377/735] 10.222.0.72 () {56 vars in 1135 bytes} [Mon Jul 11 14:26:45 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:45.598646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:45.665559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:45.712467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917be50d-7ad3-47c7-9496-d08b1938d43c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.730580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1597a06-b673-471a-bc24-b67275ce0e3c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.762553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:45.763394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.763678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.763953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.773125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:45.773841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:45.774391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:45.774988 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:45.775472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:45.775682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:45.776828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:45.779364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:45.792128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:45.793161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:45.794099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 359/736] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:26:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:45.794884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:45.801949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 378/737] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:26:45 2022] POST /identity/v3/services => generated 251 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:26:45.803666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:45.841914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:45.960291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb4bef7-5e00-4e74-9dff-c51e9c3e3d8f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:45.964561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:46.009469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:46.010905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.011425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.011912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.013001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:46.013549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:46.015287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:46.039603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 379/738] 10.222.0.72 () {56 vars in 1149 bytes} [Mon Jul 11 14:26:45 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:46.045087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:46.060796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1baaac92-a0ea-4bf2-8853-a7e681d27845 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:46.106385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb20eab6-1569-4def-8834-bc5e1cae8328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:46.107145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb20eab6-1569-4def-8834-bc5e1cae8328 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.107429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb20eab6-1569-4def-8834-bc5e1cae8328 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.107707 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb20eab6-1569-4def-8834-bc5e1cae8328 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.192812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3ecc53-105e-4df5-a7fd-3b2bf05d9356 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:46.200247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb20eab6-1569-4def-8834-bc5e1cae8328 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:46.202104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb20eab6-1569-4def-8834-bc5e1cae8328 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:46.232020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:46.241507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1be2ba-2607-4dd8-ae20-3ad11e45f693 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:46.242177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1be2ba-2607-4dd8-ae20-3ad11e45f693 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.242506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1be2ba-2607-4dd8-ae20-3ad11e45f693 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.242789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1be2ba-2607-4dd8-ae20-3ad11e45f693 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.329837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 360/739] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:45 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 535 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:46.335994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d687d2-d4da-4c19-a937-8010f605c093 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.336634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d687d2-d4da-4c19-a937-8010f605c093 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.337150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d687d2-d4da-4c19-a937-8010f605c093 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.339271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 361/740] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:46 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:46.341772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d687d2-d4da-4c19-a937-8010f605c093 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:46.417927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:46.462767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d687d2-d4da-4c19-a937-8010f605c093 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:46.471835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1be2ba-2607-4dd8-ae20-3ad11e45f693 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:46.474361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1be2ba-2607-4dd8-ae20-3ad11e45f693 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:46.487516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:46.506599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e07e8-092c-44a1-a3c7-9131b8c8a007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:46.507406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e07e8-092c-44a1-a3c7-9131b8c8a007 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.507709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e07e8-092c-44a1-a3c7-9131b8c8a007 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.507983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e07e8-092c-44a1-a3c7-9131b8c8a007 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.520061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e07e8-092c-44a1-a3c7-9131b8c8a007 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:46.521536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e07e8-092c-44a1-a3c7-9131b8c8a007 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-b68e07e8-092c-44a1-a3c7-9131b8c8a007 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:46.540623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:46.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:46.543110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:46.543110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:46.543110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:46.543110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:46.543110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 362/741] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:46.543467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d9fed9-aa93-4860-901d-8b94b163e187 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.543735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d9fed9-aa93-4860-901d-8b94b163e187 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.544015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d9fed9-aa93-4860-901d-8b94b163e187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.561486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b9d9fed9-aa93-4860-901d-8b94b163e187 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:46.562989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b9d9fed9-aa93-4860-901d-8b94b163e187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:46.603896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 380/742] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:46 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 554 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 11 14:26:46.603896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db802f45-7837-4150-bcd1-4376726331ae None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.603896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db802f45-7837-4150-bcd1-4376726331ae None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.603896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db802f45-7837-4150-bcd1-4376726331ae None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.603896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 381/743] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:46.603896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db802f45-7837-4150-bcd1-4376726331ae None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:46.641403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 363/744] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:46 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:46.642309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d9fed9-aa93-4860-901d-8b94b163e187 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:46.687612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:46.738188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db802f45-7837-4150-bcd1-4376726331ae None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:46.759737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d9fed9-aa93-4860-901d-8b94b163e187 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:46.768784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:46.776111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b208001f-1d00-4d0d-9755-4676359ef417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:46.777250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b208001f-1d00-4d0d-9755-4676359ef417 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.777554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b208001f-1d00-4d0d-9755-4676359ef417 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.777738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b208001f-1d00-4d0d-9755-4676359ef417 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.778414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b208001f-1d00-4d0d-9755-4676359ef417 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:46.778746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b208001f-1d00-4d0d-9755-4676359ef417 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:46.780108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b208001f-1d00-4d0d-9755-4676359ef417 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:46.797478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 382/745] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:46.797478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1616c052-b995-4fa7-b358-6e9c6940407a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.797478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1616c052-b995-4fa7-b358-6e9c6940407a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.797478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1616c052-b995-4fa7-b358-6e9c6940407a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.801958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:46.802837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:46.803131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:46.803468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:46.812890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1616c052-b995-4fa7-b358-6e9c6940407a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:46.814067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1616c052-b995-4fa7-b358-6e9c6940407a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:46.814188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:46.815693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:46.835483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:46.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:46.837377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:46.837377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:46.837377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:46.837377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:46.841269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 364/746] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:46.842100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:46.894470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 383/747] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:46 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 11 14:26:46.894470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1616c052-b995-4fa7-b358-6e9c6940407a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:46.945489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:46.972263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56d9b1f-70e9-4769-9dba-8130372ce62f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.009421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1616c052-b995-4fa7-b358-6e9c6940407a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.010242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.010945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.011231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.011515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.012241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.013612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.034470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 365/748] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:26:46 2022] POST /identity/v3/services => generated 240 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:47.034470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.062512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:47.070798 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.072818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.073124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.073573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.087817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.090993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:47.111651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:47.112623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:47.113505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:47.114649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 384/749] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:47.115416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.172049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39949bd8-a7ff-4a02-b64c-1a4baaee48c7 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.212826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.213502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.213786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.214067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.215085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:47.216242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.216806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.232627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 366/750] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:47.233623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.244763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405017c3-871e-4c7c-8b9d-e9988bd0a911 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.257008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:47.292614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.293820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.294158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.294498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.310111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:47.320910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:47.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:47.322736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:47.323395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 385/751] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:47.324161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.341422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:47.367208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0e0d8-ecf4-433c-ab6c-43778c7ada05 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.415672 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.416713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.417208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.417693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.418719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:47.419269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.421481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.439346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 367/752] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:47.441192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.447105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f12cf6-6353-4aae-b4c4-e06fb403b195 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.493220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.494017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.494296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.494596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.502076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.503548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:47.519062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:47.520053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:47.520967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 386/753] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:47.522072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.556416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:47.596004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e1f6da-7659-48bd-b655-3685a2a13b22 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.634380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.635266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebc80f5-3899-40d4-9e2e-604c4b6f2dd4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.635688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.636178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.636760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.637856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:47.638397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.640012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.646239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:47.653096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 368/754] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:47.654239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.678061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.678904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.679256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.679989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.686463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=055512e4e3744833ac6a9e61d22f0b63) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:47.686463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.686463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.689343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 387/755] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/role_assignments?user.id=055512e4e3744833ac6a9e61d22f0b63&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:47.690264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.778948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca39222-01fe-444c-9a4a-b8e5b545c3b6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.794909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:47.807406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1619e413-9ce1-41f0-9c08-7577ab4609e4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:47.850450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.851284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.851566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.851840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.857735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85006105-791f-443c-af09-e212afd5456b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:47.858451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85006105-791f-443c-af09-e212afd5456b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:47.858745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85006105-791f-443c-af09-e212afd5456b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:47.859025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85006105-791f-443c-af09-e212afd5456b None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:47.860178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85006105-791f-443c-af09-e212afd5456b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:47.860502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85006105-791f-443c-af09-e212afd5456b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.862180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85006105-791f-443c-af09-e212afd5456b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.864556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:47.866600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:47.889624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:47.891734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:47.893858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:47.893858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:47.893858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:47.893858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:47.894371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 369/757] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:47.896782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 388/757] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:47.896976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85006105-791f-443c-af09-e212afd5456b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.897783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:47.996431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:48.043091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85006105-791f-443c-af09-e212afd5456b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:48.049868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e9532-8bdd-471b-addb-758968ac695e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:48.092382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:48.093436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.093917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.094515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.095572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:48.096078 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:48.097855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:48.114453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:48.114453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.114453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.114832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.116921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=647a67b450184f48b3d035fe875be47c) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:48.116921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:48.118617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:48.127659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 370/758] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:48.129719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 389/759] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:47 2022] GET /identity/v3/role_assignments?user.id=647a67b450184f48b3d035fe875be47c&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 259 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:48.134458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:48.311077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae50c33-885b-4305-a365-36b9e645e613 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:48.344968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:48.361021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb4c6fd-a3d9-4cfc-b282-e592871791f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:48.362041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4c6fd-a3d9-4cfc-b282-e592871791f7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.362408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4c6fd-a3d9-4cfc-b282-e592871791f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.362851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4c6fd-a3d9-4cfc-b282-e592871791f7 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.379739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb4c6fd-a3d9-4cfc-b282-e592871791f7 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:48.381783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb4c6fd-a3d9-4cfc-b282-e592871791f7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-3eb4c6fd-a3d9-4cfc-b282-e592871791f7 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:48.397874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:48.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:48.399976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:48.402719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 371/760] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:26:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:48.408981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:48.423179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:48.424643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb6a974-ca9c-439a-b415-7922fe2892c4 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.425270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb6a974-ca9c-439a-b415-7922fe2892c4 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.425521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb6a974-ca9c-439a-b415-7922fe2892c4 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.427756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 372/761] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:48.434044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47274566-666e-4f71-9448-b1b3ebf8dafe None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.434340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47274566-666e-4f71-9448-b1b3ebf8dafe None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.434649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47274566-666e-4f71-9448-b1b3ebf8dafe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.455565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-47274566-666e-4f71-9448-b1b3ebf8dafe None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:48.456863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-47274566-666e-4f71-9448-b1b3ebf8dafe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:48.564487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6a5e42-57d0-443f-8513-7cde07e596dc None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:48.590523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 373/762] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:26:48.605928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:48.606586 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.606877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.607157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.608004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:48.608317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:48.609752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:48.628131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47274566-666e-4f71-9448-b1b3ebf8dafe None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:48.628716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 390/763] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:26:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:48.634038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:48.756910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:48.795973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab06a0d-af9c-49c2-9713-53f9b5e103f8 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:48.836446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:48.837259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.837539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.837870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.848671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:48.850217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:48.870763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:48.871952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:48.873374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:48.873374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:48.873374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:48.873374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:48.873374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 391/764] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:48.878270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:48.879184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47274566-666e-4f71-9448-b1b3ebf8dafe None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:48.908542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:48.976212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:48.977053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:48.977345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:48.977624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:48.978296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:48.979716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.000029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf897cfc-2671-45e0-942d-1392b7f0e301 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:49.011018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:49.012072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:49.012928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 374/765] 10.222.0.72 () {56 vars in 1129 bytes} [Mon Jul 11 14:26:48 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 385 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:49.019098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.024232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:49.044680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.045812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.046345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.046839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.047778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:49.048283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.049958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.065076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 392/766] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:49.071373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.186912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8a3f55-37b4-42fb-afe8-372dcb52f5c5 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.238769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.241109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.241582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.242050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.243156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:49.243523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.244954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.261667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97b7c4-7c6f-40cc-b562-0d76a3ada7d8 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.267272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 375/767] 10.222.0.72 () {56 vars in 1124 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:49.286361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.307027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.307939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.308275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.308558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.316651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.318639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:49.337380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:49.338350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:49.339293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 393/768] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:49.344623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.374832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:49.395190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:49.455024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9b2cc7-1f62-4db0-83fc-851e37b81379 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.467131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d4b80a-b3ca-45ad-8837-4b5980fa7fe5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.501668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9b6585-9e07-493a-b437-5165b9aa0d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.502793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9b6585-9e07-493a-b437-5165b9aa0d73 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.503187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9b6585-9e07-493a-b437-5165b9aa0d73 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.503491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9b6585-9e07-493a-b437-5165b9aa0d73 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.504897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9b6585-9e07-493a-b437-5165b9aa0d73 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:49.505312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9b6585-9e07-493a-b437-5165b9aa0d73 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.507851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9b6585-9e07-493a-b437-5165b9aa0d73 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.508737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e396061e-6fd5-4cd5-81b3-cddca63e3ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.509415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e396061e-6fd5-4cd5-81b3-cddca63e3ad1 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.509700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e396061e-6fd5-4cd5-81b3-cddca63e3ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.510094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e396061e-6fd5-4cd5-81b3-cddca63e3ad1 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.510948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e396061e-6fd5-4cd5-81b3-cddca63e3ad1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:49.511262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e396061e-6fd5-4cd5-81b3-cddca63e3ad1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.512809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e396061e-6fd5-4cd5-81b3-cddca63e3ad1 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.518994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 376/769] 10.222.0.72 () {56 vars in 1124 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:49.520633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69a8e3f-1c15-4c3b-a4e5-428608dcff34 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.520933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69a8e3f-1c15-4c3b-a4e5-428608dcff34 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.521739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69a8e3f-1c15-4c3b-a4e5-428608dcff34 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.523517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 377/770] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:49.525812 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69a8e3f-1c15-4c3b-a4e5-428608dcff34 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.531416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 394/771] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:49.532728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d3c1b5-94ea-4b7d-a953-a12d67387759 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.533020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d3c1b5-94ea-4b7d-a953-a12d67387759 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.533301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d3c1b5-94ea-4b7d-a953-a12d67387759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.557036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f6d3c1b5-94ea-4b7d-a953-a12d67387759 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:49.558500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f6d3c1b5-94ea-4b7d-a953-a12d67387759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:49.634462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 395/772] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:26:49.635357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d3c1b5-94ea-4b7d-a953-a12d67387759 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.645995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69a8e3f-1c15-4c3b-a4e5-428608dcff34 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.666656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:49.686646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:49.707491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.708098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.708676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.708970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.709832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] RBAC: Adding query filter params (interface=public, service_id=cca28d5d0b064a2189a0bec24d15a474, region_id=RegionOne) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:49.710301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.711743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.725340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 378/773] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/endpoints?service_id=cca28d5d0b064a2189a0bec24d15a474&interface=public®ion_id=RegionOne => generated 194 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:49.725340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.784816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d3c1b5-94ea-4b7d-a953-a12d67387759 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.831035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.832899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.833353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.834045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/2165eacee8954dce975ba81ba8ac4232/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.837136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=2165eacee8954dce975ba81ba8ac4232, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.839484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.840864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b407a-0ef5-4024-b384-79799482c222 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:49.864723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 396/774] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:49 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/2165eacee8954dce975ba81ba8ac4232/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:49.864723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:49.910925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:49.912365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:49.912945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:49.913226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] PATH_INFO: `/v3/services/network` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:49.913893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:49.916483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:49.941840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:49.942935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:49.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:49.953829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 379/775] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/services/network => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:49.955453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:50.022973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:50.046596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:50.065377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec37c16-100e-4b50-8a85-e49ebaf2dfa1 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.102875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f391872-1692-4c65-846e-1067dd5ddbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.103544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f391872-1692-4c65-846e-1067dd5ddbcd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.103829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f391872-1692-4c65-846e-1067dd5ddbcd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.104111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f391872-1692-4c65-846e-1067dd5ddbcd None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.104799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f391872-1692-4c65-846e-1067dd5ddbcd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:50.106189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f391872-1692-4c65-846e-1067dd5ddbcd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:50.118455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 397/776] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/services => generated 772 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:50.132505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff24e04-2f3e-47b7-98dc-50bab10a57bd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.161113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbfa3c6-f147-4f93-b04a-d61e84ae9f1f None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.161465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbfa3c6-f147-4f93-b04a-d61e84ae9f1f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.161750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbfa3c6-f147-4f93-b04a-d61e84ae9f1f None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.167485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 398/777] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:50.169287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e06a59-2616-4313-bdff-d0f9b4599c69 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.169705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e06a59-2616-4313-bdff-d0f9b4599c69 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.169992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e06a59-2616-4313-bdff-d0f9b4599c69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.183333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.184051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.184340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.184622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.185354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] RBAC: Adding query filter params (name=network) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:50.185670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:50.187202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:50.189810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-68e06a59-2616-4313-bdff-d0f9b4599c69 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:50.191065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-68e06a59-2616-4313-bdff-d0f9b4599c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:50.192862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 380/778] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:26:49 2022] GET /identity/v3/services?name=network => generated 124 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:50.200288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:50.281936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 399/779] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:26:50.300432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e06a59-2616-4313-bdff-d0f9b4599c69 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:50.373913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:50.383960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5816dc-137e-4b7b-9ff1-64ab42dce12d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.390488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:50.411155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e06a59-2616-4313-bdff-d0f9b4599c69 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.441547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.442243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.442561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.442850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.444113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] RBAC: Adding query filter params (type=network) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:50.444427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:50.446038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:50.457421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.458556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.459918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.462064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 381/780] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:26:50 2022] GET /identity/v3/services?type=network => generated 350 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:50.462634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.468997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:50.487862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:50.500432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:50.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:50.502317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:50.511330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 400/781] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:50.515539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:50.659793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddf2b19-4238-43a5-b2d7-64d18524cb26 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.668790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340bf4d0-83b2-4126-aadb-e2ab64c1b41d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.706459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0c765c-0b14-437a-84f6-7503ce89e6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.706744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.707484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.707694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c765c-0b14-437a-84f6-7503ce89e6c9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.707821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.708044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.708394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c765c-0b14-437a-84f6-7503ce89e6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.708844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:50.708991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c765c-0b14-437a-84f6-7503ce89e6c9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.709156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:50.710056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0c765c-0b14-437a-84f6-7503ce89e6c9 None admin] RBAC: Adding query filter params (interface=public, service_id=4b18712eb4ef4c3dbea1b8531ec280e4, region_id=RegionOne) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:50.710559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:50.710671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0c765c-0b14-437a-84f6-7503ce89e6c9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:50.712153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0c765c-0b14-437a-84f6-7503ce89e6c9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:50.718083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 382/782] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:26:50 2022] GET /identity/v3/endpoints?service_id=4b18712eb4ef4c3dbea1b8531ec280e4&interface=public®ion_id=RegionOne => generated 194 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:50.719776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d5d69-7ad5-4701-8257-825f06bc8957 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.720300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d5d69-7ad5-4701-8257-825f06bc8957 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.725535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 401/783] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:50.725535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:50.725938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d5d69-7ad5-4701-8257-825f06bc8957 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.727192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:50.728109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 383/784] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:50 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:50.729547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d5d69-7ad5-4701-8257-825f06bc8957 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:50.757491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:50.860329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d5d69-7ad5-4701-8257-825f06bc8957 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.860528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361c7915-8876-4816-9ade-cf03e02943df None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:50.902268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.903115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.903404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.903688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.903900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1929926-add4-4a47-9a92-a7a088d105c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:50.904789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1929926-add4-4a47-9a92-a7a088d105c9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.905270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1929926-add4-4a47-9a92-a7a088d105c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.905741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1929926-add4-4a47-9a92-a7a088d105c9 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.906644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1929926-add4-4a47-9a92-a7a088d105c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:50.908247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1929926-add4-4a47-9a92-a7a088d105c9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:50.915843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:50.917320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:50.917585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 402/785] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:26:50 2022] GET /identity/v3/services => generated 772 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:50.919241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e88923b-7e53-4ee0-a148-0024217eecad None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:50.919711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e88923b-7e53-4ee0-a148-0024217eecad None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:50.920148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e88923b-7e53-4ee0-a148-0024217eecad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:50.938014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:50.939067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:50.940252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:50.940252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:50.940252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:50.940252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:50.940252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 384/786] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:50.941356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5e88923b-7e53-4ee0-a148-0024217eecad None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:50.942759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5e88923b-7e53-4ee0-a148-0024217eecad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:50.945471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.042458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:51.044290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:51.055201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 403/787] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:26:51.067832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d45e43-1ace-493d-a9e1-f32ffab004d7 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.100636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e88923b-7e53-4ee0-a148-0024217eecad None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.114453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.114453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.114453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.114453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.115416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:51.115416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:51.115416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:51.128145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 385/788] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:51.134347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.283196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e88923b-7e53-4ee0-a148-0024217eecad None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.285657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78042920-c24a-4fb2-a99b-d3df0414692d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.328277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.329877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.330253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.330655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] PATH_INFO: `/v3/users/swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.331709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.332897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:51.333842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.335299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.335592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.341837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:51.343338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:51.350128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:51.358410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:51.359369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:51.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:51.361274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:51.362394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:51.368493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:51.369291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 386/790] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:51.369385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 404/790] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:51.369476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.379481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.478280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6ee314-532f-4475-a78e-f04cb6e4f4db None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.491084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:51.514954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.515692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.516042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.516386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.517298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:51.517679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:51.519210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:51.538179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 387/791] 10.222.0.72 () {56 vars in 1135 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:51.543525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.562510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1015b473-e3f0-476d-81a3-e08b11af6655 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.600976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:51.627327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.628336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.628336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.628672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.629353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:51.629972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:51.631168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:51.651010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 405/792] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:51.655735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07737b67-a064-4443-8027-8035adc97ac9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.659856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.698459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.698459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.698459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.698459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.704505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:51.705982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:51.724515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:51.725485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:51.726923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:51.726923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:51.726923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:51.726923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:51.726923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 388/793] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:51.731578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.836518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cc9003-feac-4111-9d62-dfba748fa94b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.839244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:51.843755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aecfdf8-c153-4cd7-b5bb-ffd7abb85ed9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:51.867746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:51.897938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.898674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:51.899056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.899445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.899823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.900034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:51.900483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:51.900971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:51.901845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:51.902302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:51.903852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:51.928383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 389/794] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:51.929431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:51.930996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:51.934001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:51.971691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:51.972654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:51.973574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:51.973574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:51.973574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:51.973574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:51.973574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 406/795] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:51.979028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.093446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abff4e5e-88ee-48f2-a169-853986069fb3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.156891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f006def-a0e3-47ba-847e-da3dde49ade0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.159483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.160290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.160570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.160848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.175602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:52.177220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:52.196868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.197812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.198252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.198703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.199572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:52.200294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:52.201811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:52.204690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:52.205561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:52.206481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:52.216224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 390/796] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:52.217346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 407/797] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:52.223337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.223520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.258734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:52.288751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:52.353768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1df379-41fd-4345-838e-fe1fddbdebb8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.367691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90a3ccd-94d1-469e-9419-db1bbd09274e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.391196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653f256a-e70c-4b7d-abf2-c8c5fe8f3fc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.391867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f256a-e70c-4b7d-abf2-c8c5fe8f3fc6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.392151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f256a-e70c-4b7d-abf2-c8c5fe8f3fc6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.392432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f256a-e70c-4b7d-abf2-c8c5fe8f3fc6 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.393330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653f256a-e70c-4b7d-abf2-c8c5fe8f3fc6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:52.393739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653f256a-e70c-4b7d-abf2-c8c5fe8f3fc6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:52.395201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653f256a-e70c-4b7d-abf2-c8c5fe8f3fc6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:52.411452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b80a85-a5fd-4902-84d4-516126d657a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.412857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b80a85-a5fd-4902-84d4-516126d657a0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.413409 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b80a85-a5fd-4902-84d4-516126d657a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.413515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 391/798] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:52.413617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b80a85-a5fd-4902-84d4-516126d657a0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.415206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0985ed0-1f7d-4159-9398-ef50b1c8c653 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.415683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0985ed0-1f7d-4159-9398-ef50b1c8c653 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.416025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0985ed0-1f7d-4159-9398-ef50b1c8c653 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.417956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 392/799] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:52.418801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0985ed0-1f7d-4159-9398-ef50b1c8c653 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.421545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b80a85-a5fd-4902-84d4-516126d657a0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:52.423123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b80a85-a5fd-4902-84d4-516126d657a0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-84b80a85-a5fd-4902-84d4-516126d657a0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:52.440319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:52.441342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:26:52.442300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:52.446265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 408/800] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:26:52 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:52.447727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67375fe6-2ad0-4311-ba21-5ab1a015efc7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.448121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67375fe6-2ad0-4311-ba21-5ab1a015efc7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.448410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67375fe6-2ad0-4311-ba21-5ab1a015efc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.467749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-67375fe6-2ad0-4311-ba21-5ab1a015efc7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:52.468993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-67375fe6-2ad0-4311-ba21-5ab1a015efc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:52.521024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:52.544246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0985ed0-1f7d-4159-9398-ef50b1c8c653 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.548758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 409/801] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:26:52.550352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67375fe6-2ad0-4311-ba21-5ab1a015efc7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.590807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:52.623459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.625018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.625350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.625711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/055512e4e3744833ac6a9e61d22f0b63/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.629205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=055512e4e3744833ac6a9e61d22f0b63, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:52.632143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:52.658659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 393/802] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:52 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/055512e4e3744833ac6a9e61d22f0b63/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:52.666454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.697971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67375fe6-2ad0-4311-ba21-5ab1a015efc7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.745272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906e35ac-d7f2-4971-9763-025821e4f83b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.746160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906e35ac-d7f2-4971-9763-025821e4f83b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.746609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906e35ac-d7f2-4971-9763-025821e4f83b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.746956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906e35ac-d7f2-4971-9763-025821e4f83b None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.747943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906e35ac-d7f2-4971-9763-025821e4f83b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:52.748329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906e35ac-d7f2-4971-9763-025821e4f83b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:52.749877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906e35ac-d7f2-4971-9763-025821e4f83b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:52.774588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 410/803] 10.222.0.72 () {56 vars in 1149 bytes} [Mon Jul 11 14:26:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:52.776222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5d6060-84c5-4f02-be8b-73b1e830cce5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.778401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5d6060-84c5-4f02-be8b-73b1e830cce5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.778856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5d6060-84c5-4f02-be8b-73b1e830cce5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.780702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 411/804] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:52.781613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5d6060-84c5-4f02-be8b-73b1e830cce5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.790289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a43181-61ef-49ef-9967-cde173b7ea07 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.832198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb54f6d-ac30-43d3-81c9-09839546328a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.833192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb54f6d-ac30-43d3-81c9-09839546328a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.833612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb54f6d-ac30-43d3-81c9-09839546328a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.834004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb54f6d-ac30-43d3-81c9-09839546328a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.848273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb54f6d-ac30-43d3-81c9-09839546328a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-6bb54f6d-ac30-43d3-81c9-09839546328a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:52.856634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:52.857575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:52.858411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 394/805] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:52.863709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc19a6f-ad40-4d6f-bf4a-536e1a0f6e7b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.864003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc19a6f-ad40-4d6f-bf4a-536e1a0f6e7b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.864287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc19a6f-ad40-4d6f-bf4a-536e1a0f6e7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.873629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:52.885278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:52.885727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6dc19a6f-ad40-4d6f-bf4a-536e1a0f6e7b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:52.887561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6dc19a6f-ad40-4d6f-bf4a-536e1a0f6e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:52.908312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5d6060-84c5-4f02-be8b-73b1e830cce5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:52.946364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:52.947348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:52.947725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:52.948118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:52.959670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:52.961343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:52.981153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:52.982679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:52.984594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:52.984594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:52.984594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:52.984594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:52.990225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 395/807] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:26:52.990225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc19a6f-ad40-4d6f-bf4a-536e1a0f6e7b None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:52.990628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 412/807] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:52.990628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.130153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9b9374-c47d-4c59-9c74-ead914d14dc5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.135827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc19a6f-ad40-4d6f-bf4a-536e1a0f6e7b None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.162529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:53.169968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.171120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.171407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.171688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.172444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:53.172754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.174779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.187068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.187853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.188214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.188567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.189429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:53.189830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.190381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 413/808] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:53.191273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.195882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.197834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:53.212497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 396/809] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:53.214457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.326103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea0587d-246e-4cbc-97e7-aafc50f5447f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.339333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722b5d2f-6ab3-4f5d-af02-fd2045f25575 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.377797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.379022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.379447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.379615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.380031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.380642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.380970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.381389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.381519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.383104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.389755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.391483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:53.403022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:53.404290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 11 14:26:53.405341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:53.413745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:53.414779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:53.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:53.416744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 414/811] 10.222.0.72 () {56 vars in 1129 bytes} [Mon Jul 11 14:26:53 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:53.417026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 397/811] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:53.418092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.418467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.493372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:53.495239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:53.558560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e4e8d9-141e-4e60-956e-52476be44fb1 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.564575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae282802-3815-4fe0-8d01-baff9dc4cafe None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.601898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2512ee-d083-4924-a566-9b61612a2298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.603112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2512ee-d083-4924-a566-9b61612a2298 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.603273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2512ee-d083-4924-a566-9b61612a2298 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.603506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2512ee-d083-4924-a566-9b61612a2298 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.612084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.613340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.613889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.614498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.615862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:53.616273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.616577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2512ee-d083-4924-a566-9b61612a2298 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.618217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2512ee-d083-4924-a566-9b61612a2298 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.618941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.638243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 398/812] 10.222.0.72 () {56 vars in 1124 bytes} [Mon Jul 11 14:26:53 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:53.638243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-bc2512ee-d083-4924-a566-9b61612a2298 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:53.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:53.639994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:53.640901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:53.640901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:53.640901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:53.640901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:53.642108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 415/813] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:53.644858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc8540c-b5e6-4393-832b-2d01160794f6 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.645414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc8540c-b5e6-4393-832b-2d01160794f6 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.645700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc8540c-b5e6-4393-832b-2d01160794f6 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.647778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 416/814] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:53.648425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc8540c-b5e6-4393-832b-2d01160794f6 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.762094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61df0a4f-40a7-4f10-b028-c187f74b37fc None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.766957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc8540c-b5e6-4393-832b-2d01160794f6 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:53.778816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:53.788158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:53.809352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.810149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.810564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.810916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.811893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:53.812216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.813866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.823376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fd61ec-76eb-47cb-8b03-089d42bb6ca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:53.824402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fd61ec-76eb-47cb-8b03-089d42bb6ca3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.825733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fd61ec-76eb-47cb-8b03-089d42bb6ca3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.826254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fd61ec-76eb-47cb-8b03-089d42bb6ca3 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.827537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fd61ec-76eb-47cb-8b03-089d42bb6ca3 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:53.828047 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fd61ec-76eb-47cb-8b03-089d42bb6ca3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:53.829155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 399/815] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:53.829955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fd61ec-76eb-47cb-8b03-089d42bb6ca3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:53.830157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.846183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 417/816] 10.222.0.72 () {56 vars in 1124 bytes} [Mon Jul 11 14:26:53 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:53.848125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f49ce-8b57-4bad-bbfb-320b1da6da19 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:53.848666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f49ce-8b57-4bad-bbfb-320b1da6da19 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:53.849410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f49ce-8b57-4bad-bbfb-320b1da6da19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:53.870654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-359f49ce-8b57-4bad-bbfb-320b1da6da19 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:53.871880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-359f49ce-8b57-4bad-bbfb-320b1da6da19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:53.950252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 418/817] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:26:53.951153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359f49ce-8b57-4bad-bbfb-320b1da6da19 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:53.970930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff367e5a-a351-468e-a22f-34b5e70136d0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.012144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.012857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.013145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.013427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.014160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:54.014498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.015954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.036351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 400/818] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:54.036351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:54.098766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:54.104147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359f49ce-8b57-4bad-bbfb-320b1da6da19 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.105730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:54.150305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.151244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.151528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.151804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/647a67b450184f48b3d035fe875be47c/roles/b032af1c9f7a447a98b75130f5bab37a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.169413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=647a67b450184f48b3d035fe875be47c, role_id=b032af1c9f7a447a98b75130f5bab37a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.170233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e2191-7414-4271-8004-cc993c85baff None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.170948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.187461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 419/819] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:26:53 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/647a67b450184f48b3d035fe875be47c/roles/b032af1c9f7a447a98b75130f5bab37a => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:26:54.188304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:54.245183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.246144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.246371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.246740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.249120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.252317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.290696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 401/820] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:26:54 2022] POST /identity/v3/endpoints => generated 346 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:26:54.292474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:54.407853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5976cc-79be-4073-8a10-983923c8f98e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.477480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.478549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.478936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.479350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] PATH_INFO: `/v3/services/network` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.480144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.481705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.482943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:54.497184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:54.498611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 11 14:26:54.499990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:54.500756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf03cf4a-99f9-48cf-abfc-cf0e095b8568 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.500832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 420/821] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:54 2022] GET /identity/v3/services/network => generated 88 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:54.511619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:54.540161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:54.561983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.563480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.563980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.564950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.579100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.580935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:54.606058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:54.607751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:26:54.608906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:54.613151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 402/822] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:26:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:54.620981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:54.682840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e491941-c6e3-494e-995d-79260edb5747 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.732135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.733070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.733255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.733525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.734212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] RBAC: Adding query filter params (name=network) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:54.734555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.736417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.745711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 421/823] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:26:54 2022] GET /identity/v3/services?name=network => generated 124 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:54.749856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:54.766151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439ce4b5-1543-44ae-bc8e-33f503ad7ff3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.820240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.822785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.823741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.824573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.826820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:54.827852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.830404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.849654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:54.861809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 403/824] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:26:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:54.868236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:54.880372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd085f5-e935-453f-ac9c-10dea492e3c0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:54.887997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:54.919941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57077691-2cb9-46e8-b2ba-0025b3e78237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:54.920624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57077691-2cb9-46e8-b2ba-0025b3e78237 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.920956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57077691-2cb9-46e8-b2ba-0025b3e78237 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.921240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57077691-2cb9-46e8-b2ba-0025b3e78237 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.922056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57077691-2cb9-46e8-b2ba-0025b3e78237 None admin] RBAC: Adding query filter params (type=network) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:54.922375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57077691-2cb9-46e8-b2ba-0025b3e78237 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:54.924072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57077691-2cb9-46e8-b2ba-0025b3e78237 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:54.930107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 422/825] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:26:54 2022] GET /identity/v3/services?type=network => generated 350 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:54.935599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59994b6b-76ac-4b5b-a5d0-f2fa5da746df None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:54.935872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59994b6b-76ac-4b5b-a5d0-f2fa5da746df None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:54.936162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59994b6b-76ac-4b5b-a5d0-f2fa5da746df None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:54.937712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 423/826] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:54 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:54.939197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59994b6b-76ac-4b5b-a5d0-f2fa5da746df None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:55.001903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c9a99c-ebed-4297-9981-9d088f14b298 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:55.046724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:55.048378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.049320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.050117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.074140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59994b6b-76ac-4b5b-a5d0-f2fa5da746df None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:55.110964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55a518f-8b51-493a-a50c-4f4aa7b9bc7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:55.111637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a518f-8b51-493a-a50c-4f4aa7b9bc7f None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.111926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a518f-8b51-493a-a50c-4f4aa7b9bc7f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.112208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a518f-8b51-493a-a50c-4f4aa7b9bc7f None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.112966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55a518f-8b51-493a-a50c-4f4aa7b9bc7f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:55.114350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55a518f-8b51-493a-a50c-4f4aa7b9bc7f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:55.125020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:55.133237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:55.135253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:55.146873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 424/827] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:26:54 2022] POST /identity/v3/endpoints => generated 327 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:26:55.148397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc2efb-f84d-4cd3-9c21-7c4643e14fee None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.148744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc2efb-f84d-4cd3-9c21-7c4643e14fee None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.149036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc2efb-f84d-4cd3-9c21-7c4643e14fee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.153929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:55.183004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5bcc2efb-f84d-4cd3-9c21-7c4643e14fee None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:55.184244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5bcc2efb-f84d-4cd3-9c21-7c4643e14fee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:55.279102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 425/828] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:55 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:26:55.319566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcc2efb-f84d-4cd3-9c21-7c4643e14fee None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:55.338045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 404/829] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:54 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 470 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:26:55.343846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:55.479466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcc2efb-f84d-4cd3-9c21-7c4643e14fee None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:55.507912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:55.523478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:55.524566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.525030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.525477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.537890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:55.546714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:55.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:26:55.548923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:55.551297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 426/830] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:55.560652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:55.635853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5346dc9c-d81e-4d73-b39f-cfc89496919d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:55.651534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:55.671963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edc481f-4db1-4297-bb50-73b2e8cb9393 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:55.675790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:55.676571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.676862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.677325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.689804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:55.691318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:55.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:55.713187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:55.714091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:55.714091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:55.714091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:55.714091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:55.714091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 405/831] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:55.722463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:55.732112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:55.734018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.734575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.735067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.736582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:55.737136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:55.739196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:55.751130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 427/832] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:26:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:55.757281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:55.807553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:55.832937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab66c25-9084-4e16-993f-529997afa8a3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:55.868262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c60be2-5091-4d05-9e9b-c48666d237f7 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:55.869351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:55.870221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.870647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.871022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.872158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:55.872339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:55.874025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:55.888515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 406/833] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:55.893326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:55.907421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:55.908577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:55.909074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:55.909656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:55.923890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:55.925674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:55.927297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:55.948823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:55.949789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:55.950998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:55.950998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:55.950998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:55.950998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:55.957822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 428/834] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:55.959156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:56.004722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74201f1b-4b49-4b83-b44a-52fc9ff7112b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:56.041772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:56.042604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.042894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.043175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.050979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:56.052968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:56.064926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:56.070088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:56.071103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:56.072012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:56.072450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a842a1-9ab7-42b2-b89f-93014c871eb0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:56.079726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 407/835] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:56.080945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:56.110085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de820a55-f967-4488-accf-8097547107a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:56.110941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de820a55-f967-4488-accf-8097547107a2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.111298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de820a55-f967-4488-accf-8097547107a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.111655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de820a55-f967-4488-accf-8097547107a2 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.112463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de820a55-f967-4488-accf-8097547107a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:56.113008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de820a55-f967-4488-accf-8097547107a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:56.114641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de820a55-f967-4488-accf-8097547107a2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:56.128473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 429/836] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:56.134025 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de820a55-f967-4488-accf-8097547107a2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:56.178575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:56.195565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34f4fd3-e1b2-4a7d-a980-787a4b9b1b1c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:56.234569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:56.235582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.236064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.236527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.237600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:56.238194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:56.240394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:56.247877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de820a55-f967-4488-accf-8097547107a2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:56.262246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 408/837] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:56.263647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:56.287358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:56.288302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.288665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.289022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] PATH_INFO: `/v3/users/swift` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.297642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:56.299444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:56.317410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:56.318484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 11 14:26:56.319477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:56.323525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 430/838] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:26:56 2022] GET /identity/v3/users/swift => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:56.324502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:56.345940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:56.384294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a466ee-7202-43ff-a74e-7a859d7dec3b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:56.423350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:56.424471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.424966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.425446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.426505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=2165eacee8954dce975ba81ba8ac4232) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:56.427119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:56.428945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:56.442461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d35e5b9-f946-472a-98e2-f39d52938b4b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:56.443708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 409/839] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:56 2022] GET /identity/v3/role_assignments?user.id=2165eacee8954dce975ba81ba8ac4232&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:56.478666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dfd912-0ae4-4daf-b66f-85586ed841c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:56.479366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dfd912-0ae4-4daf-b66f-85586ed841c4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.479658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dfd912-0ae4-4daf-b66f-85586ed841c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.479937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dfd912-0ae4-4daf-b66f-85586ed841c4 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.480803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dfd912-0ae4-4daf-b66f-85586ed841c4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:56.481119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dfd912-0ae4-4daf-b66f-85586ed841c4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:56.482739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dfd912-0ae4-4daf-b66f-85586ed841c4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:56.515100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 431/840] 10.222.0.72 () {56 vars in 1135 bytes} [Mon Jul 11 14:26:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:26:56.520928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:56.534964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:56.638453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007173f2-edae-4e2c-908e-7d040107a119 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:56.679496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc0bd2f-b009-4858-a7d8-5bdbfba2e759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:56.680182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0bd2f-b009-4858-a7d8-5bdbfba2e759 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.680475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0bd2f-b009-4858-a7d8-5bdbfba2e759 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.680939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0bd2f-b009-4858-a7d8-5bdbfba2e759 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.786650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc0bd2f-b009-4858-a7d8-5bdbfba2e759 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:56.789158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc0bd2f-b009-4858-a7d8-5bdbfba2e759 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:56.808281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:56.836588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb538822-886d-4ee3-aa54-bc02d5afec90 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.837109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb538822-886d-4ee3-aa54-bc02d5afec90 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.837221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb538822-886d-4ee3-aa54-bc02d5afec90 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.838797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 432/841] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:56.844237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c090e6-c360-4b6c-af87-bbf6c52fdd66 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.844533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c090e6-c360-4b6c-af87-bbf6c52fdd66 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.844823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c090e6-c360-4b6c-af87-bbf6c52fdd66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.862414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-83c090e6-c360-4b6c-af87-bbf6c52fdd66 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:56.863730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-83c090e6-c360-4b6c-af87-bbf6c52fdd66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:56.956777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 410/842] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:56 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:26:56.959691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d7da0-eb1f-40fa-ab08-693fec53168f None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.960308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d7da0-eb1f-40fa-ab08-693fec53168f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.960907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d7da0-eb1f-40fa-ab08-693fec53168f None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.962487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 411/843] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:26:56.967479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d7da0-eb1f-40fa-ab08-693fec53168f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:56.969473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 433/844] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:56 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:26:56.976701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ee0ca-eccd-4c86-a449-d30a54a77fa7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:56.976701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ee0ca-eccd-4c86-a449-d30a54a77fa7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:56.976701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ee0ca-eccd-4c86-a449-d30a54a77fa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:56.985331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.005663 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a23ee0ca-eccd-4c86-a449-d30a54a77fa7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:57.007370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a23ee0ca-eccd-4c86-a449-d30a54a77fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:57.111840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 434/845] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:56 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:26:57.113105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23ee0ca-eccd-4c86-a449-d30a54a77fa7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.129535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d7da0-eb1f-40fa-ab08-693fec53168f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.141923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.168426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.169232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.169513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.169792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.180702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.182168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:57.200870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:57.201981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:57.203033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:57.203033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:57.203033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:57.203033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:57.203033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 412/846] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:57.207708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.234901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23ee0ca-eccd-4c86-a449-d30a54a77fa7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.272530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.273337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.273623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.273900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.275939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.288991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:26:57.296863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:57.297805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:57.298693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 435/847] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:57.299869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.321461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e693fd-b334-4f92-b75c-defaa969990b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.362385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.363221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.363506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.363801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.364432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.365870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:57.377848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:57.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:57.385394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 413/848] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:57.386214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.412887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.434315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb4255-a92e-4c01-9a4a-33eaf7432213 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.469027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.469619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.469913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.470178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.470933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:57.471245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.472646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.489891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 436/849] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:57.489891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.495661 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa3aed-b9cf-441e-aed3-18db00537083 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.532357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.533131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.533415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.533692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.534462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:57.534759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.536188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.546476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 414/850] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:57.546476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.559238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.607160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e302ff36-3f11-4afc-816a-49746a00b9c6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.640948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.646537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.647466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.647466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.647760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.648455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:57.648754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.650132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.659798 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 437/851] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:57.660006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a052a-95e0-4e71-927a-5e86f41f046f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.660127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.702263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.703720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.703844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.704121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.713534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.715068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:57.731237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:57.732342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 415/852] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:57.733232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.779256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48f9f3-39d2-481a-9436-9f52b43de7e0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.800039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.820234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.821404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.821688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.821965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.832341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.847872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674834ba-d962-4c1e-8cad-e24502120a6d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:57.853251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:57.854480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:57.855391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:57.855391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:57.855391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:57.855391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:57.859215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 438/853] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:57.859928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.885193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:57.885964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:57.886558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:57.886693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:57.887388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:57.887657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:57.889345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:57.902463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 416/854] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:57.902463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:57.919740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:57.986521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37364d6-c1b3-46b8-920d-b8bd2f15c579 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:58.028146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:58.028990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.029344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.029687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.030588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:58.031124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:58.032963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:58.035231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34343f3f-bdcc-4933-b9c8-969841160151 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:58.047756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 439/855] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:58.053974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:58.085005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:58.085776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.086159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.086534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.087393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:58.087809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:58.089294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:58.098902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:58.109591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 417/856] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:58.110735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:58.185080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc022721-1207-4646-a915-2787ce6dd900 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:58.198270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:58.221916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:58.222957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.223423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.223878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.225437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=055512e4e3744833ac6a9e61d22f0b63) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:58.225939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:58.227603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:58.234552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 440/857] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:58 2022] GET /identity/v3/role_assignments?user.id=055512e4e3744833ac6a9e61d22f0b63&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:58.239377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b093cbd-c929-4c35-9802-552ddc932198 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:58.282498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:58.283649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.284179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.284640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.294495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:58.296355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:26:58.313739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:58.315094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:26:58.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:58.318946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 418/858] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:26:58 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:58.325095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:58.433609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:58.440294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d290409-11c8-4a51-b177-d54e3ab97e26 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:58.483246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d191e596-12f8-47bb-ac41-09b4db9a59f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:58.483936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191e596-12f8-47bb-ac41-09b4db9a59f2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.484219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191e596-12f8-47bb-ac41-09b4db9a59f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.484493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191e596-12f8-47bb-ac41-09b4db9a59f2 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.485345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d191e596-12f8-47bb-ac41-09b4db9a59f2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:58.485656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d191e596-12f8-47bb-ac41-09b4db9a59f2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:58.487262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d191e596-12f8-47bb-ac41-09b4db9a59f2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:58.507268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 441/859] 10.222.0.72 () {56 vars in 1149 bytes} [Mon Jul 11 14:26:58 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:58.512772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:58.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c703c882-ed34-4f80-b98f-2bb53de05ac7 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:58.670316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dac4c9-4ea3-4a90-8dce-6bb39c04d946 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.670699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dac4c9-4ea3-4a90-8dce-6bb39c04d946 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.670908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dac4c9-4ea3-4a90-8dce-6bb39c04d946 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.672459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 442/860] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:26:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:26:58.678190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:58.678530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f046bd-f9b3-4210-af36-7d688d3c81bd None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.679061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f046bd-f9b3-4210-af36-7d688d3c81bd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.679547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f046bd-f9b3-4210-af36-7d688d3c81bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.685286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:58.685976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.686270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.686591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.699548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-01f046bd-f9b3-4210-af36-7d688d3c81bd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:26:58.701451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-01f046bd-f9b3-4210-af36-7d688d3c81bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:26:58.764528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:58.765917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:58.806768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 443/861] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:26:58 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:26:58.820891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f046bd-f9b3-4210-af36-7d688d3c81bd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:58.876522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 419/862] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:26:58 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:26:58.882213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:58.914880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:58.917955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:58.940338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f046bd-f9b3-4210-af36-7d688d3c81bd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:58.986749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:58.987691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:58.987985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:58.988371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:58.989044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:58.990643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:58.994107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cdac46-9ff3-4cdf-ac01-a4caffcbc1fe None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:59.004163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:59.005161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:26:59.006592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 444/863] 10.222.0.72 () {56 vars in 1132 bytes} [Mon Jul 11 14:26:58 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:59.012965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:59.036920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:59.038396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:59.038934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:59.039424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:59.050557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:59.052906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:26:59.071767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:59.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:59.073660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:26:59.073660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:26:59.073660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:26:59.073660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:59.076351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 420/864] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:26:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:26:59.081661 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:59.145252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689d8431-d734-4a60-aa85-8d8e33dcf553 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:59.187674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:59.188750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:59.189251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:59.189756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:59.190714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:59.191245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:59.192901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:59.198905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 445/865] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:26:59 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:59.204523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:59.205452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:59.221084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:59.223340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309f3117-0b11-4138-8f67-90303d2378e1 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:59.263554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:59.264641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:59.265163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:59.265926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:59.267097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:59.267631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:59.269133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:59.288503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 421/866] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:26:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:59.291746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:59.326279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f572c4f-4617-4181-b941-7c0079a55c41 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:59.367787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:59.368935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:59.369484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:59.369958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:59.370896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:59.371403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:59.373094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:59.379049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 446/867] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:26:59 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:59.432441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d018f53-4ff6-4f1f-9dd0-1d1864ffa74a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:59.472440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:59.473654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:59.474912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:59.475436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:59.484029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:59.485835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:26:59.502119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:26:59.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:26:59.504012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:26:59.507254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 422/868] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:26:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:26:59.513276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:59.611618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:59.640378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717fc75b-a373-4561-9fee-31e9ac1c4dcd None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:59.681257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75322a1-ba77-4e77-aef1-ce9fd0e79c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:59.681938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75322a1-ba77-4e77-aef1-ce9fd0e79c13 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:59.682225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75322a1-ba77-4e77-aef1-ce9fd0e79c13 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:59.682528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75322a1-ba77-4e77-aef1-ce9fd0e79c13 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:59.683332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75322a1-ba77-4e77-aef1-ce9fd0e79c13 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:59.683645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75322a1-ba77-4e77-aef1-ce9fd0e79c13 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:59.685085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75322a1-ba77-4e77-aef1-ce9fd0e79c13 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:59.699126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 447/869] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:26:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:26:59.705201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:26:59.709286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:26:59.815248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d2ced1-5502-4a3a-b591-ccb02fc5e400 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:26:59.852910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fb4bb8-3b6e-4991-befd-fbefb82bb750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:26:59.853595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fb4bb8-3b6e-4991-befd-fbefb82bb750 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:26:59.853878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fb4bb8-3b6e-4991-befd-fbefb82bb750 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:26:59.854156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fb4bb8-3b6e-4991-befd-fbefb82bb750 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:26:59.855033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fb4bb8-3b6e-4991-befd-fbefb82bb750 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=647a67b450184f48b3d035fe875be47c) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:26:59.855350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fb4bb8-3b6e-4991-befd-fbefb82bb750 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:26:59.856805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fb4bb8-3b6e-4991-befd-fbefb82bb750 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:26:59.864003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 423/870] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:26:59 2022] GET /identity/v3/role_assignments?user.id=647a67b450184f48b3d035fe875be47c&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:00.105041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a0949-880f-40f1-ab22-cc7fc4efd3c4 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:00.105592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a0949-880f-40f1-ab22-cc7fc4efd3c4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:00.105884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a0949-880f-40f1-ab22-cc7fc4efd3c4 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:00.107742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 448/871] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:00.113430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d34fb8-00a7-477e-b07b-eba2ff34799a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:00.114070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d34fb8-00a7-477e-b07b-eba2ff34799a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:00.114070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d34fb8-00a7-477e-b07b-eba2ff34799a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:00.132072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-98d34fb8-00a7-477e-b07b-eba2ff34799a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:00.133283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-98d34fb8-00a7-477e-b07b-eba2ff34799a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:00.203799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:00.218545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 424/872] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:00 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:00.235374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a0949-880f-40f1-ab22-cc7fc4efd3c4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:00.353721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a0949-880f-40f1-ab22-cc7fc4efd3c4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:00.390985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c682c7cb-2aa5-416c-8f45-b84b83f87cc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:00.391581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c682c7cb-2aa5-416c-8f45-b84b83f87cc2 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:00.391849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c682c7cb-2aa5-416c-8f45-b84b83f87cc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:00.392135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c682c7cb-2aa5-416c-8f45-b84b83f87cc2 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:00.392883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c682c7cb-2aa5-416c-8f45-b84b83f87cc2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:00.394272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c682c7cb-2aa5-416c-8f45-b84b83f87cc2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:00.396583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:00.412697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 449/873] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:27:00 2022] POST /identity/v3/services => generated 251 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:00.530233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b65c2f-1d49-4d7f-9f2a-6963c0d174a2 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:00.530626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b65c2f-1d49-4d7f-9f2a-6963c0d174a2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:00.530997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b65c2f-1d49-4d7f-9f2a-6963c0d174a2 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:00.532779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 425/874] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:00.538757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfdb359-09e6-4e61-b2d6-c48bd9dd5bed None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:00.539081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfdb359-09e6-4e61-b2d6-c48bd9dd5bed None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:00.539317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfdb359-09e6-4e61-b2d6-c48bd9dd5bed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:00.558398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-8cfdb359-09e6-4e61-b2d6-c48bd9dd5bed None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:00.559683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8cfdb359-09e6-4e61-b2d6-c48bd9dd5bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:00.649936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 450/875] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:00 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:00.672747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b65c2f-1d49-4d7f-9f2a-6963c0d174a2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:00.785984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b65c2f-1d49-4d7f-9f2a-6963c0d174a2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:00.821633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:00.822457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:00.822749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:00.823030 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:00.823689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:00.825317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:00.837801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:00.838823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:00.839667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 426/876] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:00 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:00.847809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfdb359-09e6-4e61-b2d6-c48bd9dd5bed None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:00.953404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfdb359-09e6-4e61-b2d6-c48bd9dd5bed None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:00.975007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:00.996071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1541c4f7-dc56-43a7-a408-9e92adcde01a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:00.996758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1541c4f7-dc56-43a7-a408-9e92adcde01a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:00.997045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1541c4f7-dc56-43a7-a408-9e92adcde01a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:00.997325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1541c4f7-dc56-43a7-a408-9e92adcde01a None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:00.998049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1541c4f7-dc56-43a7-a408-9e92adcde01a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:00.998363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1541c4f7-dc56-43a7-a408-9e92adcde01a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:00.999779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1541c4f7-dc56-43a7-a408-9e92adcde01a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:01.005785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 451/877] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:00 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:01.011484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:01.077184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:01.119132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec215dd-e127-4dfc-882d-f4013d32a042 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:01.155237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec3aacb-eef4-48f0-ba5d-d22e30990230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:01.156042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3aacb-eef4-48f0-ba5d-d22e30990230 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:01.156345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3aacb-eef4-48f0-ba5d-d22e30990230 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:01.156627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3aacb-eef4-48f0-ba5d-d22e30990230 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:01.157435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec3aacb-eef4-48f0-ba5d-d22e30990230 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:01.157750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec3aacb-eef4-48f0-ba5d-d22e30990230 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:01.159159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec3aacb-eef4-48f0-ba5d-d22e30990230 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:01.165252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 427/878] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:01 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:01.578991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fde2369-8baa-4fe8-a570-723ea77c6c8b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:01.579528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fde2369-8baa-4fe8-a570-723ea77c6c8b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:01.579528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fde2369-8baa-4fe8-a570-723ea77c6c8b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:01.581196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 452/879] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:01.587375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b2ea62-d513-4b96-b7e0-65fa2962d261 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:01.587734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b2ea62-d513-4b96-b7e0-65fa2962d261 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:01.588026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b2ea62-d513-4b96-b7e0-65fa2962d261 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:01.605497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-02b2ea62-d513-4b96-b7e0-65fa2962d261 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:01.606734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-02b2ea62-d513-4b96-b7e0-65fa2962d261 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:01.684035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 428/880] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:01 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:01.706900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fde2369-8baa-4fe8-a570-723ea77c6c8b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:01.816735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fde2369-8baa-4fe8-a570-723ea77c6c8b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:01.853901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b33ffa2-d52e-4932-aae0-aa218a306338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:01.854518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b33ffa2-d52e-4932-aae0-aa218a306338 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:01.854805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b33ffa2-d52e-4932-aae0-aa218a306338 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:01.855086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b33ffa2-d52e-4932-aae0-aa218a306338 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:01.855881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b33ffa2-d52e-4932-aae0-aa218a306338 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:01.857313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b33ffa2-d52e-4932-aae0-aa218a306338 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:01.875118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 453/881] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:27:01 2022] POST /identity/v3/services => generated 269 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:02.193774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf915d-cb01-4d39-8082-c1a34223911f None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:02.194369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf915d-cb01-4d39-8082-c1a34223911f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:02.194601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf915d-cb01-4d39-8082-c1a34223911f None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:02.196149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 429/882] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:02.203927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac0f7f-148f-41d6-9f97-35d54a49b55a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:02.203927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac0f7f-148f-41d6-9f97-35d54a49b55a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:02.203927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac0f7f-148f-41d6-9f97-35d54a49b55a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:02.222670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c8ac0f7f-148f-41d6-9f97-35d54a49b55a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:02.224772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c8ac0f7f-148f-41d6-9f97-35d54a49b55a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:02.267052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:02.310792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 454/883] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:02 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:02.329315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf915d-cb01-4d39-8082-c1a34223911f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:02.392874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:02.441727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf915d-cb01-4d39-8082-c1a34223911f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:02.478985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:02.479775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:02.480060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:02.480337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:02.491774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:02.500213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:02.501235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:02.502291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 430/884] 10.222.0.72 () {56 vars in 1120 bytes} [Mon Jul 11 14:27:02 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:02.507839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac0f7f-148f-41d6-9f97-35d54a49b55a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:02.620341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac0f7f-148f-41d6-9f97-35d54a49b55a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:02.663989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9e672c-0ca8-48d2-bf4c-5ebbb5073454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:02.664749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e672c-0ca8-48d2-bf4c-5ebbb5073454 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:02.665038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e672c-0ca8-48d2-bf4c-5ebbb5073454 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:02.665312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e672c-0ca8-48d2-bf4c-5ebbb5073454 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:02.666084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e672c-0ca8-48d2-bf4c-5ebbb5073454 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:02.666391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e672c-0ca8-48d2-bf4c-5ebbb5073454 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:02.667810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e672c-0ca8-48d2-bf4c-5ebbb5073454 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:02.677965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 455/885] 10.222.0.72 () {56 vars in 1115 bytes} [Mon Jul 11 14:27:02 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:02.687560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:02.719527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9dc328-5e2c-4d82-88d7-61b720a8ab8e None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:02.719869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9dc328-5e2c-4d82-88d7-61b720a8ab8e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:02.720104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9dc328-5e2c-4d82-88d7-61b720a8ab8e None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:02.722846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 456/886] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:02.727754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6381cfc-bd32-477b-bc3b-573b0faca3bc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:02.728054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6381cfc-bd32-477b-bc3b-573b0faca3bc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:02.728340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6381cfc-bd32-477b-bc3b-573b0faca3bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:02.747554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f6381cfc-bd32-477b-bc3b-573b0faca3bc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:02.749083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f6381cfc-bd32-477b-bc3b-573b0faca3bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:02.789198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:02.799825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aea0c3f-4ae5-44be-9211-47b5430036a8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:02.818223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:02.845636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:02.846476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:02.846758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:02.847039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:02.856392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 457/887] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:02 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:02.860796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:02.862711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:02.880095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6381cfc-bd32-477b-bc3b-573b0faca3bc None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:02.884887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:02.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:02.886978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:02.886978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:02.886978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:02.886978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:02.886978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 431/888] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:02.891735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.011788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320dc7c-aab7-47cf-b3cd-f62b766d8f7c None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.013987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6381cfc-bd32-477b-bc3b-573b0faca3bc None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.052635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.053309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.053591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.053866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.054616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:03.054926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.056300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.064303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.065416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.066369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.067162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.068616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.070231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.074279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 432/889] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:03.075481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:03.086175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:03.087506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:27:03.088578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:03.096593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 458/890] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:27:02 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:03.098068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.117520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:03.133881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:03.198461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2172d0-8c9f-4870-acd5-e4ea650de286 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.215839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7637f1-79c7-4b24-90ff-a51c8d438816 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.236346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.237322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.237707 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.238082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.246024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.247685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.256310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fd2cc8-0f4e-4f60-ba77-8ba2df534d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.257271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd2cc8-0f4e-4f60-ba77-8ba2df534d07 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.257709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd2cc8-0f4e-4f60-ba77-8ba2df534d07 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.258139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd2cc8-0f4e-4f60-ba77-8ba2df534d07 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.259144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fd2cc8-0f4e-4f60-ba77-8ba2df534d07 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:03.259574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fd2cc8-0f4e-4f60-ba77-8ba2df534d07 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.260966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fd2cc8-0f4e-4f60-ba77-8ba2df534d07 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:27:03.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:03.266539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:27:03.267511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:03.270833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 459/891] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:27:03 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:03.271494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 433/892] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:27:03 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:03.272359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.272712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d8fb4-0e1b-4932-b096-bd05d4532dae None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.273127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d8fb4-0e1b-4932-b096-bd05d4532dae None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.273468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d8fb4-0e1b-4932-b096-bd05d4532dae None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.274940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 460/893] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:03.278660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0d8fb4-0e1b-4932-b096-bd05d4532dae None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.385308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:03.387164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:03.402199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e86a8c3-b809-40b3-8636-8a23f8186478 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.408270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0d8fb4-0e1b-4932-b096-bd05d4532dae None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.439039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad4a3a2-ce02-43ec-85db-f8b5e48584ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.439720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad4a3a2-ce02-43ec-85db-f8b5e48584ca None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.440004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad4a3a2-ce02-43ec-85db-f8b5e48584ca None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.440288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad4a3a2-ce02-43ec-85db-f8b5e48584ca None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.441009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad4a3a2-ce02-43ec-85db-f8b5e48584ca None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:03.441323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad4a3a2-ce02-43ec-85db-f8b5e48584ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.443266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad4a3a2-ce02-43ec-85db-f8b5e48584ca None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.449082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 434/894] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:27:03 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:03.452275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179d59a-cd50-4f20-b20f-a4d3dd74a04e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.453295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179d59a-cd50-4f20-b20f-a4d3dd74a04e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.454009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179d59a-cd50-4f20-b20f-a4d3dd74a04e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.465119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.465900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.466184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.466582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.467619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:03.467822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.469266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.471739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d179d59a-cd50-4f20-b20f-a4d3dd74a04e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:03.473105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d179d59a-cd50-4f20-b20f-a4d3dd74a04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:03.499249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 461/895] 10.222.0.72 () {56 vars in 1149 bytes} [Mon Jul 11 14:27:03 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:03.499249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.574626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 435/896] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:03 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:03.574626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179d59a-cd50-4f20-b20f-a4d3dd74a04e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.642143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb09718e-65c8-48b0-9eda-eb93fff5a585 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.688826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:03.690031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.690812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.691202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.691408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.692269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] RBAC: Adding query filter params (interface=public, service_id=9861900cd0d445629c5e493d4807bfc1, region_id=RegionOne) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:03.692564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.694478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.704031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 462/897] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:27:03 2022] GET /identity/v3/endpoints?service_id=9861900cd0d445629c5e493d4807bfc1&interface=public®ion_id=RegionOne => generated 194 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:03.704653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.728448 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179d59a-cd50-4f20-b20f-a4d3dd74a04e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.729025 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:03.777352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.778016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.778308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.778618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.838445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff99950-30b2-441e-b1a7-3883dd9afdbf None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:03.859704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.861172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.876221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:03.876906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:03.877195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:03.877476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:03.878207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:03.879628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:03.895867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 463/898] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:27:03 2022] POST /identity/v3/services => generated 241 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:03.897114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:03.986221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.004021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.017378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 436/899] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:03 2022] GET /identity/v3/auth/tokens => generated 1362 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:04.017809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.074713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ac1fb-83b5-4e45-a473-bd8466652d57 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.119789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4394e7-6671-42dd-a59e-35ef40b81173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.120512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4394e7-6671-42dd-a59e-35ef40b81173 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.120864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4394e7-6671-42dd-a59e-35ef40b81173 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.121220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4394e7-6671-42dd-a59e-35ef40b81173 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.121989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4394e7-6671-42dd-a59e-35ef40b81173 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.123490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4394e7-6671-42dd-a59e-35ef40b81173 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.129748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 464/900] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:27:03 2022] GET /identity/v3/services => generated 1497 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:27:04.131220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676f1a37-9e52-4f35-bb80-bc2cf64a8b07 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.131585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676f1a37-9e52-4f35-bb80-bc2cf64a8b07 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.131870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676f1a37-9e52-4f35-bb80-bc2cf64a8b07 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.134479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 465/901] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:04.138841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b4da4-9e64-425c-843d-46f378c4cdfa None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.139408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b4da4-9e64-425c-843d-46f378c4cdfa None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.139867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b4da4-9e64-425c-843d-46f378c4cdfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.151571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec3f11-d7bb-42e0-abc0-d18f99334a76 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.160994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fe2b4da4-9e64-425c-843d-46f378c4cdfa None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:04.163312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fe2b4da4-9e64-425c-843d-46f378c4cdfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:04.191032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.191944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.192324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.193214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.206596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.208180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:04.228196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:04.229208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:04.230181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:04.230181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:04.230181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:04.230181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:04.234545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 437/902] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:04.235648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.271786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 466/903] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:04 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:04.278162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.294294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2b4da4-9e64-425c-843d-46f378c4cdfa None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.376756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42284df-2713-4bde-8d5c-26569839d766 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.386062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.431259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.432111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.432499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.432787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.433525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:04.433996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.435888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.438177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2b4da4-9e64-425c-843d-46f378c4cdfa None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.451305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 438/904] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:04.456785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.481538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.482784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.483406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.483842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.484803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.486569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:04.503137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:04.504316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:27:04.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:04.509452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 467/905] 10.222.0.72 () {56 vars in 1132 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:04.516357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.589107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.601129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed40424-cb19-4554-accb-46df1f4311e3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.648322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.649448 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.649721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.650099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.658657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.660323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.662511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7970d362-7b5b-453f-9673-e10fd988f6a5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:27:04.677815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:04.678992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:04.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:04.686821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 439/906] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:04.687951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.701808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.714298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.715329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.715785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.716279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.717412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:04.717963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.719708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.726659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 468/907] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:04.735878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.816653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1606c7e4-7ecc-4139-aa2e-9e664627b6a2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.856729 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.857396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.857680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.857956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.858775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:04.859177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.860625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.868221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00d0475-8eb7-4887-a8cf-14b9286ddbaa None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:04.873986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 440/908] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:04.879471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.897255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.910447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:04.911064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:04.911351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:04.912122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:04.912853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:04.913162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:04.914572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:04.924905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 469/909] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:04.925877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:04.985953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:04.992692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f52295-4887-43a3-b993-abfb047ecc53 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:05.029353 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:05.030019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:05.030308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:05.030617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:05.031452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] RBAC: Adding query filter params (role.id=3e10326cb64249db9b62814df9290687, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=647a67b450184f48b3d035fe875be47c) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:05.031828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:05.033305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:05.038199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd894e7f-9460-4447-9589-d879c341e7b4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:05.042472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 441/910] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/role_assignments?user.id=647a67b450184f48b3d035fe875be47c&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=3e10326cb64249db9b62814df9290687 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:05.089682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7ee1f7-4bad-4ff3-a92e-4fdf8665a417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:05.090369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7ee1f7-4bad-4ff3-a92e-4fdf8665a417 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:05.090684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7ee1f7-4bad-4ff3-a92e-4fdf8665a417 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:05.090965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7ee1f7-4bad-4ff3-a92e-4fdf8665a417 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:05.091922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7ee1f7-4bad-4ff3-a92e-4fdf8665a417 None admin] RBAC: Adding query filter params (interface=public, service_id=9a0bcaf5a7be451ab8638536b31d3bec, region_id=RegionOne) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:05.092393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7ee1f7-4bad-4ff3-a92e-4fdf8665a417 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:05.094142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7ee1f7-4bad-4ff3-a92e-4fdf8665a417 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:05.100881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 470/911] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:27:04 2022] GET /identity/v3/endpoints?service_id=9a0bcaf5a7be451ab8638536b31d3bec&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:05.106691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:05.197871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:05.225018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa501c42-9715-4bf8-8131-bac6379cfa88 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:05.261355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adc94a0-7062-4d20-9f61-f059de8b0c8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:05.262052 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adc94a0-7062-4d20-9f61-f059de8b0c8f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:05.262340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adc94a0-7062-4d20-9f61-f059de8b0c8f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:05.262668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adc94a0-7062-4d20-9f61-f059de8b0c8f None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:05.263513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adc94a0-7062-4d20-9f61-f059de8b0c8f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:05.264957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adc94a0-7062-4d20-9f61-f059de8b0c8f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:05.273710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 442/912] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:27:05 2022] GET /identity/v3/services => generated 1497 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:27:06.073952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb253b40-ede1-446a-81e6-84d26bcb5d93 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.074526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb253b40-ede1-446a-81e6-84d26bcb5d93 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.074526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb253b40-ede1-446a-81e6-84d26bcb5d93 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.076408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 471/913] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:06.081830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb452f-e067-4b47-a4c5-77c30cbe4308 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.082128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb452f-e067-4b47-a4c5-77c30cbe4308 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.082409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb452f-e067-4b47-a4c5-77c30cbe4308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.100090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b4eb452f-e067-4b47-a4c5-77c30cbe4308 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:06.101284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b4eb452f-e067-4b47-a4c5-77c30cbe4308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:06.172342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 443/914] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:06 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:06.195584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb253b40-ede1-446a-81e6-84d26bcb5d93 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.318228 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb253b40-ede1-446a-81e6-84d26bcb5d93 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:06.337254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:06.356163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:06.356986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.357273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.357548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.358208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:06.359634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:06.367779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf45f1d-e750-49c7-a367-a6e83fc9a8f6 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.368249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf45f1d-e750-49c7-a367-a6e83fc9a8f6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.368351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf45f1d-e750-49c7-a367-a6e83fc9a8f6 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.370947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 444/915] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:06.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:06.373073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:06.374549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 472/916] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:06.376008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf692e3c-dce8-41db-a382-34cca5cebb36 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.376008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf692e3c-dce8-41db-a382-34cca5cebb36 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.376008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf692e3c-dce8-41db-a382-34cca5cebb36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.383132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.399261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cf692e3c-dce8-41db-a382-34cca5cebb36 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:06.400575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cf692e3c-dce8-41db-a382-34cca5cebb36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:06.478606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 445/917] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:06 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:06.500405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a155ed1-ab1e-470e-8bdd-1525cb656aa0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:06.501385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf692e3c-dce8-41db-a382-34cca5cebb36 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.503706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:06.543653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:06.544332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.544620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.544900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.545624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:06.545937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:06.547406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:06.553402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 473/918] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:06.560426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.623214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf692e3c-dce8-41db-a382-34cca5cebb36 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:06.623854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:06.663800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:06.665202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.665489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.665764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.666751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:06.670913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:06.683821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ae54f1-31ab-482f-8d73-3fff1288bddc None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:06.689830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:06.690911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 11 14:27:06.691781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:06.695323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 446/919] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:06.710452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.729431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:06.730211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.730547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.730835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.731552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:06.731988 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:06.733582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:06.740230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 474/920] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:06.745937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.774511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:06.812311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21500c7-6f0f-4470-b32e-8e048e4a64d4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:06.849332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:06.850127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.850509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.850859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.851656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:06.852038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:06.853538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:06.859038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 447/921] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:06.861858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de05788-9708-40fc-86ba-0333f93143e2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:06.864380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.876699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:06.913245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:06.914298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:06.914876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:06.915357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:06.916503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] RBAC: Adding query filter params (interface=public, service_id=c90eee73ab3e484ea684416a449fbfcd, region_id=RegionOne) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:06.917064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:06.918688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:06.929263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 475/922] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/endpoints?service_id=c90eee73ab3e484ea684416a449fbfcd&interface=public®ion_id=RegionOne => generated 194 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:06.930594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:06.977932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad90c48-227b-4589-9df2-8921ddda2076 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.008412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:07.017473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.018463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.018747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.019028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.019780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:07.020373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:07.022051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:07.027695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 448/923] 10.222.0.72 () {56 vars in 1114 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:07.033119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.045937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a514c8-f88a-4f87-917a-bd6108a3fad0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.084069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6753c657-bea5-4bb3-baa6-24cbb7bcb9d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.085547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6753c657-bea5-4bb3-baa6-24cbb7bcb9d1 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.086060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6753c657-bea5-4bb3-baa6-24cbb7bcb9d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.086690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6753c657-bea5-4bb3-baa6-24cbb7bcb9d1 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.087702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6753c657-bea5-4bb3-baa6-24cbb7bcb9d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:07.089302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6753c657-bea5-4bb3-baa6-24cbb7bcb9d1 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:07.102537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 476/924] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:27:06 2022] GET /identity/v3/services => generated 1497 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:27:07.146910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c523cb6-c5c0-4389-a691-45c752474983 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.188040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.188759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.189054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.189342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.190248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:07.191753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:07.202602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e69cb4-78bc-4b5f-a5a3-31974da4bad7 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.203237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e69cb4-78bc-4b5f-a5a3-31974da4bad7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.203734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e69cb4-78bc-4b5f-a5a3-31974da4bad7 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.205633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 477/925] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:27:07.212380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aad36a-fb54-45b7-949d-b5658a0512d8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.212726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aad36a-fb54-45b7-949d-b5658a0512d8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.212951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aad36a-fb54-45b7-949d-b5658a0512d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.221481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 449/926] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:27:07 2022] POST /identity/v3/endpoints => generated 346 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:07.234546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d1aad36a-fb54-45b7-949d-b5658a0512d8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:07.236411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d1aad36a-fb54-45b7-949d-b5658a0512d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:07.277982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:07.324556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 478/927] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:07 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:07.348030 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.409905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:07.437587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05431760-7583-4029-8b0d-a1d7665fda28 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.437908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05431760-7583-4029-8b0d-a1d7665fda28 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.438161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05431760-7583-4029-8b0d-a1d7665fda28 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.439869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 479/928] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:07.446267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b578dee-91b0-468e-8fff-81365390ab34 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.446267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b578dee-91b0-468e-8fff-81365390ab34 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.446267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b578dee-91b0-468e-8fff-81365390ab34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.467742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5b578dee-91b0-468e-8fff-81365390ab34 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:07.469735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5b578dee-91b0-468e-8fff-81365390ab34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:07.476209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a046c3a0-7454-4036-b662-e4d46576a6c6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.518144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.519034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.519307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.519589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.531557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:07.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:07.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:07.543336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 450/929] 10.222.0.72 () {56 vars in 1120 bytes} [Mon Jul 11 14:27:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:07.549082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.573015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 480/930] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:07 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:07.597201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b578dee-91b0-468e-8fff-81365390ab34 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.672124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29d4a36-c9df-4b9e-9149-8fee5cbb9d4b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.683955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:07.699455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:07.708435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b578dee-91b0-468e-8fff-81365390ab34 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.710412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.711122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.711408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.711691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.712675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:07.712786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:07.714129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:07.728368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 451/931] 10.222.0.72 () {56 vars in 1115 bytes} [Mon Jul 11 14:27:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:07.729291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.749669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.750991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.751488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.752005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.752892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:07.754580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:07.768341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:07.769360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 11 14:27:07.770261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:07.777606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 481/932] 10.222.0.72 () {56 vars in 1132 bytes} [Mon Jul 11 14:27:07 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:07.778770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.837419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af164c0-cc9f-4e7b-944f-b8efdbb32b58 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.873980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.874826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.875094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.875377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.885389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102232ba-96cf-492e-b721-5737f78265e6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:07.887474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:07.888955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:07.906840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:07.907981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:07.909082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:07.909082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:07.909082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:07.909082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:07.913293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 452/933] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:07.914168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.924709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:07.925622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:07.926055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:07.926506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:07.927455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:07.927930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:07.929550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:07.934762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 482/934] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:07 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:07.941968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:07.947089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:07.986879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:08.050915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefc7076-0875-4ad1-b582-84b86338cc82 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:08.076490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c24574-d972-45d8-8757-3ed61cc03aa6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:08.098460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:08.099231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:08.099519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:08.099801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:08.100542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:08.100838 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:08.102317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:08.120010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 453/935] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:08.121781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:08.124807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:08.127092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:08.127616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:08.128115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:08.129072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:08.129607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:08.131852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:08.140126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 483/936] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:07 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:08.146307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:08.241841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:08.247456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:08.264885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775426f5-9e52-46cc-8978-616859dbd075 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:08.271797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50558452-c5c3-496a-a90e-d27ff8068866 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:08.303084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:08.303940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:08.304229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:08.304508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:08.313121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:08.314646 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:08.323015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:08.324210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:08.324707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:08.325456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:08.326410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:08.328020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:27:08.330988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:08.331921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 454/937] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:27:08 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:08.338341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:08.372226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 484/938] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:27:08 2022] POST /identity/v3/endpoints => generated 347 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:08.477571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a442a133-2b61-4b08-8c2e-2dbcd8472c72 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:08.526142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:08.526782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:08.527129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:08.527412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:08.528327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:08.528648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:08.530141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:08.541377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:08.552235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 455/939] 10.222.0.72 () {56 vars in 1149 bytes} [Mon Jul 11 14:27:08 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:08.558415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:08.682644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ea398-7b71-4c35-8939-15e3cd059a98 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:08.722051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:08.722913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:08.723198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:08.723500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:08.735951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:08.737436 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:08.739450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:08.759729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:08.760699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:08.761600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:08.761600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:08.761600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:08.761600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:08.761600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 485/940] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:08.767029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:08.902379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4819feb-74cc-41ca-b80d-1388f2e5e661 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:08.940177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:08.941317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:08.941812 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:08.942294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:08.943344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:08.943870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:08.945511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:08.959876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 456/941] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:08.966487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:09.078324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e30da0-48d7-4381-baef-d1e1d6e24cb0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:09.116269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a69ce5c-2e69-4928-961f-5a872de7dd0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:09.117087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a69ce5c-2e69-4928-961f-5a872de7dd0a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.117375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a69ce5c-2e69-4928-961f-5a872de7dd0a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.117661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a69ce5c-2e69-4928-961f-5a872de7dd0a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.125654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a69ce5c-2e69-4928-961f-5a872de7dd0a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:09.127191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a69ce5c-2e69-4928-961f-5a872de7dd0a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-1a69ce5c-2e69-4928-961f-5a872de7dd0a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:27:09.142743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:09.143694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:09.144590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 486/942] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:09.152220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:09.227008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:09.280806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4825f74-a538-4b74-8638-537b02020752 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.281130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4825f74-a538-4b74-8638-537b02020752 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.281397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4825f74-a538-4b74-8638-537b02020752 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.283311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 487/943] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:27:09.289290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c637ccf-da00-44d6-8bab-320f02451611 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.289590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c637ccf-da00-44d6-8bab-320f02451611 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.289876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c637ccf-da00-44d6-8bab-320f02451611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.299611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507e8b30-3348-447e-8f1e-16feefd071c0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:09.310855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-3c637ccf-da00-44d6-8bab-320f02451611 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:09.312528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3c637ccf-da00-44d6-8bab-320f02451611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:09.337664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:09.338763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.339296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.339776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.340780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:09.341309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:09.343046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:09.362549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 457/944] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:27:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:09.367011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:09.383159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:09.404930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 488/945] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:09 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:09.430048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c637ccf-da00-44d6-8bab-320f02451611 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:09.492779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd287dbe-e0ae-4895-a2b1-34f602f05608 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:09.513763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:09.535682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dfa9f-aa14-4675-8230-a5140a02e030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:09.537328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dfa9f-aa14-4675-8230-a5140a02e030 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.537616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dfa9f-aa14-4675-8230-a5140a02e030 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.537899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dfa9f-aa14-4675-8230-a5140a02e030 None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/647a67b450184f48b3d035fe875be47c/roles/3e10326cb64249db9b62814df9290687` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.543323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9dfa9f-aa14-4675-8230-a5140a02e030 None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=647a67b450184f48b3d035fe875be47c, role_id=3e10326cb64249db9b62814df9290687)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:09.544832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9dfa9f-aa14-4675-8230-a5140a02e030 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:09.555766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c637ccf-da00-44d6-8bab-320f02451611 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:09.569436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 458/946] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:27:09 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/647a67b450184f48b3d035fe875be47c/roles/3e10326cb64249db9b62814df9290687 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:27:09.569436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9282928-035f-4046-ba43-d834e7516909 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.569436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9282928-035f-4046-ba43-d834e7516909 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.569436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9282928-035f-4046-ba43-d834e7516909 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.569436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 459/947] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:09.571037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfb6bb-f88c-4c86-9f1e-cbc402d8c905 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.571404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfb6bb-f88c-4c86-9f1e-cbc402d8c905 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.571760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfb6bb-f88c-4c86-9f1e-cbc402d8c905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.591987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d4cfb6bb-f88c-4c86-9f1e-cbc402d8c905 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:09.593360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d4cfb6bb-f88c-4c86-9f1e-cbc402d8c905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:09.612886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:09.614057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.614621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.615077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.616059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:09.617724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:09.631877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:09.632973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:09.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:09.636951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 489/948] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:09 2022] GET /identity/v3/services/object-store => generated 93 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:09.643619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:09.696984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 460/949] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:09 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:09.721009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cfb6bb-f88c-4c86-9f1e-cbc402d8c905 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:09.725814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:09.788947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e09b9f-9367-4c41-b361-8b00120844a2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:09.851246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:09.851934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.852225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.852508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.853232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:09.853549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:09.854984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:09.866212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 490/950] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:09 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:09.872789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:09.875412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cfb6bb-f88c-4c86-9f1e-cbc402d8c905 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:09.891367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:09.956115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:09.957177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:09.957754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:09.958807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:09.960600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:09.963207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:09.964921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING keystone.server.flask.common [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 11 14:27:09.990977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.sql.core [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 11 14:27:09.990977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 11 14:27:09.990977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [parameters: {'id': 'a1e8280d92604ccaa99aba8ad7732b11', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 11 14:27:09.990977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80423) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id a1e8280d92604ccaa99aba8ad7732b11.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id a1e8280d92604ccaa99aba8ad7732b11. Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result = self._query(query) Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application conn.query(q) Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 11 14:27:10.002289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result.read() Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 11 14:27:10.003384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application next(self.gen) Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application yield resource Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application next(self.gen) Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.session.rollback() Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.force_reraise() Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise self.value Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application session.commit() Jul 11 14:27:10.004348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._prepare_impl() Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.session.flush() Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._flush(objects) Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application compat.raise_( Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application flush_context.execute() Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 11 14:27:10.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rec.execute(self) Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 11 14:27:10.007916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result = self._query(query) Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application conn.query(q) Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application result.read() Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 11 14:27:10.009857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [parameters: {'id': 'a1e8280d92604ccaa99aba8ad7732b11', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:10.012430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:10.014368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id a1e8280d92604ccaa99aba8ad7732b11. Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 461/951] 10.222.0.72 () {60 vars in 1140 bytes} [Mon Jul 11 14:27:09 2022] POST /identity/v3/projects => generated 262 bytes in 284 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:10.018016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:10.044091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee3fb4-8b20-4a1f-9654-3a69d1f74040 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:10.070771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:10.098815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:10.100437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:10.100952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:10.101442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:10.102378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:10.102923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:10.105056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:10.120244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 491/952] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:09 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:10.121580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:10.175410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48675e50-c28a-4d97-bbcd-f63cea4bfbbd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:10.237983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:10.239590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:10.240102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:10.240675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:10.249755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:10.254046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:10.277811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d5a493-09a1-41ac-9f0a-edd199d4d93e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:27:10.282721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:10.284191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:10.285714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:10.289692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 462/953] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:10.297951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:10.338929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:10.363578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f8918c-1282-4b46-a614-31abdfc93995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:10.365193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f8918c-1282-4b46-a614-31abdfc93995 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:10.365565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f8918c-1282-4b46-a614-31abdfc93995 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:10.365928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f8918c-1282-4b46-a614-31abdfc93995 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:10.366868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f8918c-1282-4b46-a614-31abdfc93995 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:10.369151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f8918c-1282-4b46-a614-31abdfc93995 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:10.398483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 492/954] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:27:10 2022] POST /identity/v3/endpoints => generated 349 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:10.465000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e093e45-b6bb-4178-b1d5-a10434f4afcc None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:10.528494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938e51cc-421d-445b-aaa6-72754e34c3c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:10.529908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e51cc-421d-445b-aaa6-72754e34c3c0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:10.530346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e51cc-421d-445b-aaa6-72754e34c3c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:10.530882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e51cc-421d-445b-aaa6-72754e34c3c0 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:10.531985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938e51cc-421d-445b-aaa6-72754e34c3c0 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:10.532522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938e51cc-421d-445b-aaa6-72754e34c3c0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:10.534270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938e51cc-421d-445b-aaa6-72754e34c3c0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:10.566768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 463/955] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:27:10 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:10.694050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c5163d-6f04-44d1-8267-d9722faac6a4 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:10.694343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c5163d-6f04-44d1-8267-d9722faac6a4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:10.694666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c5163d-6f04-44d1-8267-d9722faac6a4 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:10.696381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 493/956] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:27:10.702584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f3106-40fe-4250-b34e-55860d33d4f6 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:10.702978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f3106-40fe-4250-b34e-55860d33d4f6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:10.703276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f3106-40fe-4250-b34e-55860d33d4f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:10.727454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-847f3106-40fe-4250-b34e-55860d33d4f6 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:10.729432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-847f3106-40fe-4250-b34e-55860d33d4f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:10.811511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 464/957] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:10 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:10.835375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c5163d-6f04-44d1-8267-d9722faac6a4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:10.873033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:10.955666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c5163d-6f04-44d1-8267-d9722faac6a4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:10.999393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:11.000262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:11.000544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:11.000821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] PATH_INFO: `/v3/services/compute` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:11.001805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:11.003308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:11.016859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:11.018022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:11.018920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 494/958] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:10 2022] GET /identity/v3/services/compute => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:11.026378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847f3106-40fe-4250-b34e-55860d33d4f6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:11.051023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:11.136449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847f3106-40fe-4250-b34e-55860d33d4f6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:11.173900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd7530b-7dd9-4341-a2e2-26facd0a15b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:11.174702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd7530b-7dd9-4341-a2e2-26facd0a15b4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:11.174994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd7530b-7dd9-4341-a2e2-26facd0a15b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:11.175266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd7530b-7dd9-4341-a2e2-26facd0a15b4 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:11.176192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd7530b-7dd9-4341-a2e2-26facd0a15b4 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:11.176518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd7530b-7dd9-4341-a2e2-26facd0a15b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:11.178645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd7530b-7dd9-4341-a2e2-26facd0a15b4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:11.184450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 465/959] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:27:11 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:11.191635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:11.297328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:11.304667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3cd63-925f-4288-987f-e820b2a4863d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:11.341786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356a72d2-d0ff-4268-a651-abaab9a3cb31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:11.342479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a72d2-d0ff-4268-a651-abaab9a3cb31 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:11.342768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a72d2-d0ff-4268-a651-abaab9a3cb31 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:11.343048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a72d2-d0ff-4268-a651-abaab9a3cb31 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:11.343761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356a72d2-d0ff-4268-a651-abaab9a3cb31 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:11.344066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356a72d2-d0ff-4268-a651-abaab9a3cb31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:11.345763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356a72d2-d0ff-4268-a651-abaab9a3cb31 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:11.351422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 495/960] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:27:11 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:12.007103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4d20c-0035-47cc-9ba5-a231ec42d3d8 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.007643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4d20c-0035-47cc-9ba5-a231ec42d3d8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.007643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4d20c-0035-47cc-9ba5-a231ec42d3d8 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.009429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 466/961] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:12.016650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916c604-aa8a-4898-a3c6-ae965be509a3 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.017089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916c604-aa8a-4898-a3c6-ae965be509a3 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.017468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916c604-aa8a-4898-a3c6-ae965be509a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.037270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-3916c604-aa8a-4898-a3c6-ae965be509a3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:12.038553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3916c604-aa8a-4898-a3c6-ae965be509a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:12.119749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 496/962] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:12 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:12.143248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4d20c-0035-47cc-9ba5-a231ec42d3d8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:12.234095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:12.264015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4d20c-0035-47cc-9ba5-a231ec42d3d8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:12.307578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:12.308474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.308762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.309157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.324867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:12.334629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:12.336881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:12.338911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 467/963] 10.222.0.72 () {56 vars in 1120 bytes} [Mon Jul 11 14:27:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:12.343757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916c604-aa8a-4898-a3c6-ae965be509a3 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:12.442259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:12.454998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916c604-aa8a-4898-a3c6-ae965be509a3 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:12.517114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:12.518189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.518726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.519215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.520716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:12.521267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:12.523955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:12.539948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 497/964] 10.222.0.72 () {56 vars in 1115 bytes} [Mon Jul 11 14:27:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:12.545726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:12.666235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7532c8-d9a7-46f3-8f4e-219bf63f6a00 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:12.710018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:12.710904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.711186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.711464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.713636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:12.725299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:12.726943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:12.748854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:12.751229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:12.753380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:12.753380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:12.753380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:12.753380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:12.753380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 468/965] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:12.757689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:12.887887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db01ba2-5e79-4fa9-a5a3-157c7e594acb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:12.925357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acea746-574d-4da4-ae8b-4f4aafa921c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:12.926055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acea746-574d-4da4-ae8b-4f4aafa921c5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.926338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acea746-574d-4da4-ae8b-4f4aafa921c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.926653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acea746-574d-4da4-ae8b-4f4aafa921c5 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.927489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acea746-574d-4da4-ae8b-4f4aafa921c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:12.927873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acea746-574d-4da4-ae8b-4f4aafa921c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:12.929297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acea746-574d-4da4-ae8b-4f4aafa921c5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:12.943911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 498/966] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:12.949514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:12.970834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e4a153-048a-415d-81d4-66252afef25b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.971306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e4a153-048a-415d-81d4-66252afef25b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.971600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e4a153-048a-415d-81d4-66252afef25b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.974856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 499/967] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:12.979286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f4b3bd-bb1c-4244-ad63-b24f2ad3d26e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:12.979585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f4b3bd-bb1c-4244-ad63-b24f2ad3d26e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:12.979881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f4b3bd-bb1c-4244-ad63-b24f2ad3d26e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:12.998534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-85f4b3bd-bb1c-4244-ad63-b24f2ad3d26e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:13.000052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-85f4b3bd-bb1c-4244-ad63-b24f2ad3d26e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:13.009546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:13.069016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d51a92-373e-492e-a3e5-bf0772bd35ee None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:13.086639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 500/968] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:12 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:13.112844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f4b3bd-bb1c-4244-ad63-b24f2ad3d26e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:13.116995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:13.118744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.119251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.119721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.129576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:13.131995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:27:13.150123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:13.151499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 11 14:27:13.152723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:13.155595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 469/969] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:27:12 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:13.162650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:13.220632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:13.246206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f4b3bd-bb1c-4244-ad63-b24f2ad3d26e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:13.287788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da12a42-0014-49f7-b83d-c26e7a1def98 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:13.290785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:13.291480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.291770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.292055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.292794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:13.294632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:13.296115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING keystone.server.flask.common [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 11 14:27:13.313655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.sql.core [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 11 14:27:13.313655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 11 14:27:13.313655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [parameters: {'user_id': '57d1143f49aa488fb38406df6a7ed816', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 11 14:27:13.313655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80424) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-a955b747-1ef6-4f8b-8f82-bf60ed7caf98 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application result = self._query(query) Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application conn.query(q) Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application result.read() Jul 11 14:27:13.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 11 14:27:13.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application next(self.gen) Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application yield resource Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application next(self.gen) Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self.session.rollback() Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self.force_reraise() Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise self.value Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application session.commit() Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 11 14:27:13.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._prepare_impl() Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self.session.flush() Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._flush(objects) Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application compat.raise_( Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application flush_context.execute() Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rec.execute(self) Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 11 14:27:13.339425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 11 14:27:13.340346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application result = self._query(query) Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application conn.query(q) Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application result.read() Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 11 14:27:13.341196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [parameters: {'user_id': '57d1143f49aa488fb38406df6a7ed816', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:13.342084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 11 14:27:13.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 11 14:27:13.344739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 11 14:27:13.344739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 11 14:27:13.344739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 11 14:27:13.344739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 11 14:27:13.344739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 11 14:27:13.344739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.346529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 501/970] 10.222.0.72 () {60 vars in 1131 bytes} [Mon Jul 11 14:27:13 2022] POST /identity/v3/users => generated 160 bytes in 235 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:13.349726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88852d3-fccb-4d76-a650-a6261555c96c None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.350221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88852d3-fccb-4d76-a650-a6261555c96c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.350758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88852d3-fccb-4d76-a650-a6261555c96c None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.351202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4562891c-6892-4f25-a3db-b9eff41ca4a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:13.352413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562891c-6892-4f25-a3db-b9eff41ca4a5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.352937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562891c-6892-4f25-a3db-b9eff41ca4a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.353455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562891c-6892-4f25-a3db-b9eff41ca4a5 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.355619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4562891c-6892-4f25-a3db-b9eff41ca4a5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:13.356213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4562891c-6892-4f25-a3db-b9eff41ca4a5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:13.359081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4562891c-6892-4f25-a3db-b9eff41ca4a5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:13.360856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 502/971] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:13 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:13.362196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88852d3-fccb-4d76-a650-a6261555c96c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:13.370248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:13.392407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 470/972] 10.222.0.72 () {56 vars in 1149 bytes} [Mon Jul 11 14:27:13 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:13.399231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe81973-1d1f-48ce-a704-2b6178607a7d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.399593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe81973-1d1f-48ce-a704-2b6178607a7d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.399934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe81973-1d1f-48ce-a704-2b6178607a7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.421330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1fe81973-1d1f-48ce-a704-2b6178607a7d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:13.422967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1fe81973-1d1f-48ce-a704-2b6178607a7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:13.526195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 471/973] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:13 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:13.528045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe81973-1d1f-48ce-a704-2b6178607a7d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:13.546767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88852d3-fccb-4d76-a650-a6261555c96c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:13.570834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:13.583954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5029a4-b6c4-495f-ab2d-47e285544806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:13.584763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5029a4-b6c4-495f-ab2d-47e285544806 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.585044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5029a4-b6c4-495f-ab2d-47e285544806 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5029a4-b6c4-495f-ab2d-47e285544806 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.592763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5029a4-b6c4-495f-ab2d-47e285544806 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:13.594352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5029a4-b6c4-495f-ab2d-47e285544806 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-9c5029a4-b6c4-495f-ab2d-47e285544806 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:27:13.609914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:13.611108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.612255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 503/974] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:27:13 2022] GET /identity/v3/users/cinder => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:13.616477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33a694-ca4a-4fb0-ac29-d801a3a3ec3a None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.616895 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33a694-ca4a-4fb0-ac29-d801a3a3ec3a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.617318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33a694-ca4a-4fb0-ac29-d801a3a3ec3a None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.620743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 504/975] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:13 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:13.620743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff33a694-ca4a-4fb0-ac29-d801a3a3ec3a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:13.641674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe81973-1d1f-48ce-a704-2b6178607a7d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:13.684289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98fee93-58e8-4100-8c05-a6de051ea954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:13.684961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98fee93-58e8-4100-8c05-a6de051ea954 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.685250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98fee93-58e8-4100-8c05-a6de051ea954 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.685533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98fee93-58e8-4100-8c05-a6de051ea954 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.691330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:13.736850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff33a694-ca4a-4fb0-ac29-d801a3a3ec3a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:13.760239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98fee93-58e8-4100-8c05-a6de051ea954 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:13.761701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98fee93-58e8-4100-8c05-a6de051ea954 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:13.774621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:13.775434 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.775719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.775992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.776658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:13.778055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:13.790670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:13.797215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:13.798267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 505/976] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:13.799941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:13.800026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:13.867311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 472/977] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:13 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:13.868643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c57285-de71-476f-ae5e-3fec8e313fc5 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.868937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c57285-de71-476f-ae5e-3fec8e313fc5 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.869221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c57285-de71-476f-ae5e-3fec8e313fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.885961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-07c57285-de71-476f-ae5e-3fec8e313fc5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:13.887218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-07c57285-de71-476f-ae5e-3fec8e313fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:13.898110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c797da-769b-4c25-a017-3cb3251baabe None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:13.926267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:13.934817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:13.935504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:13.935794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:13.936080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:13.936897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:13.937212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:13.938704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:13.957330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 473/978] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:13 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:13.958843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c57285-de71-476f-ae5e-3fec8e313fc5 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:13.977495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 506/979] 10.222.0.72 () {56 vars in 1101 bytes} [Mon Jul 11 14:27:13 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:13.978184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.055115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:14.102179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c57285-de71-476f-ae5e-3fec8e313fc5 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.138967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c104141e-f1be-41df-a3df-4654c4b47d04 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.149287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.149975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.150688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.150688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.151690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:14.151690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.152964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.158647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 474/980] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:13 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:14.159497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.181010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.181744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.182036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.182320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.197441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:14.221190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:14.227559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:14.228666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:14.228666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:14.228666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:14.228666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:14.228666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 507/981] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:14.228666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.268973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:14.280429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b079f13-fc83-4a37-b01a-f83f14daa1f1 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.309989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:14.317967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.318667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.318956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.319233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.319958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.321326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.334032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a692b89f-0b8f-40db-abd9-b78d9c71c65c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.336538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 475/982] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:27:14 2022] POST /identity/v3/services => generated 242 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:14.337361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.384040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.384906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.385198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.385475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.386213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:14.386563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.388017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.394697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 508/983] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:14 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:14.401405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.468944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63bc30-4e96-449b-8dd9-0a4bd45c3206 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.507976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.508654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.509370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.509370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.509985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:14.510301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.511731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.514247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:14.521306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa516ab-dd4e-4058-9ddf-07f90967814b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.528942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 476/984] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:14.535852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.563476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.564201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.564485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.564758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.565631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] RBAC: Adding query filter params (interface=admin, service_id=c90eee73ab3e484ea684416a449fbfcd, region_id=RegionOne) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:14.565952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.567991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.575423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 509/985] 10.222.0.72 () {56 vars in 1249 bytes} [Mon Jul 11 14:27:14 2022] GET /identity/v3/endpoints?service_id=c90eee73ab3e484ea684416a449fbfcd&interface=admin®ion_id=RegionOne => generated 193 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:14.584128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.608125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:14.680138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4db68ea-d867-4ef1-8a05-f0002f07a1fc None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.725278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.726004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.726289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.726598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.734156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.734394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1d8b8c-ff19-4dbf-95b4-a2a4919239e2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.735680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:27:14.750706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:14.751879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:14.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:14.757130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 477/986] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:14.758012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:14.776119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.777142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.777647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.778124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.779119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.780895 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.788691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 510/987] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:27:14 2022] GET /identity/v3/services => generated 1727 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:27:14.909745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ce516b-ecc0-43bd-8f51-84a1398ab1b6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:14.921819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:14.955457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991cae16-1ab8-4db9-8937-85724e7cac85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:14.956568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991cae16-1ab8-4db9-8937-85724e7cac85 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:14.956989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991cae16-1ab8-4db9-8937-85724e7cac85 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:14.957450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991cae16-1ab8-4db9-8937-85724e7cac85 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:14.959451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991cae16-1ab8-4db9-8937-85724e7cac85 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:14.959998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991cae16-1ab8-4db9-8937-85724e7cac85 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:14.962482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991cae16-1ab8-4db9-8937-85724e7cac85 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:14.983426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 478/988] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:27:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:14.990262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:15.021941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:15.121295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60682793-9762-4e8e-af0c-464c31875b2e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:15.175773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc9a461-e5d7-407a-b0af-c34b11d82e4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:15.176721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc9a461-e5d7-407a-b0af-c34b11d82e4c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:15.177166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc9a461-e5d7-407a-b0af-c34b11d82e4c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:15.177617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc9a461-e5d7-407a-b0af-c34b11d82e4c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:15.179137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc9a461-e5d7-407a-b0af-c34b11d82e4c None admin] RBAC: Adding query filter params (role.id=3e10326cb64249db9b62814df9290687, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=647a67b450184f48b3d035fe875be47c) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:15.179599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc9a461-e5d7-407a-b0af-c34b11d82e4c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:15.181766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc9a461-e5d7-407a-b0af-c34b11d82e4c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:15.192203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 511/989] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:14 2022] GET /identity/v3/role_assignments?user.id=647a67b450184f48b3d035fe875be47c&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=3e10326cb64249db9b62814df9290687 => generated 608 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:16.548045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459c928f-f6c8-4494-8f8d-118e966972a8 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:16.548608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459c928f-f6c8-4494-8f8d-118e966972a8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:16.548608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459c928f-f6c8-4494-8f8d-118e966972a8 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:16.550021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 479/990] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:16.556522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04a800-1f73-4d61-a1de-a5f51b4c0fb7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:16.557352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04a800-1f73-4d61-a1de-a5f51b4c0fb7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:16.557352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04a800-1f73-4d61-a1de-a5f51b4c0fb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:16.575971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-7e04a800-1f73-4d61-a1de-a5f51b4c0fb7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:16.577547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7e04a800-1f73-4d61-a1de-a5f51b4c0fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:16.654508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 512/991] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:16 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:16.678840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459c928f-f6c8-4494-8f8d-118e966972a8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:16.866542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459c928f-f6c8-4494-8f8d-118e966972a8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:16.930456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9528bee3-d6e0-4498-ac64-98ddff51b1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:16.930456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9528bee3-d6e0-4498-ac64-98ddff51b1b9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:16.930456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9528bee3-d6e0-4498-ac64-98ddff51b1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:16.930893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9528bee3-d6e0-4498-ac64-98ddff51b1b9 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:16.934453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9528bee3-d6e0-4498-ac64-98ddff51b1b9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:16.934662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9528bee3-d6e0-4498-ac64-98ddff51b1b9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-9528bee3-d6e0-4498-ac64-98ddff51b1b9 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:16.961698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:16.962795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:16.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 480/992] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:16 2022] GET /identity/v3/services/compute => generated 88 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:16.972636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e04a800-1f73-4d61-a1de-a5f51b4c0fb7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.014158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:17.094198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b5e92-ac4a-46e6-8a22-34683fc59409 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.094874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b5e92-ac4a-46e6-8a22-34683fc59409 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.095479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b5e92-ac4a-46e6-8a22-34683fc59409 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.098515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 481/993] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:17.108837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf053a17-2ee8-42bc-9c3b-b24244142932 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.109343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf053a17-2ee8-42bc-9c3b-b24244142932 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.109849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf053a17-2ee8-42bc-9c3b-b24244142932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.140383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cf053a17-2ee8-42bc-9c3b-b24244142932 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:17.142296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cf053a17-2ee8-42bc-9c3b-b24244142932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:17.150590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e04a800-1f73-4d61-a1de-a5f51b4c0fb7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:17.187102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:17.187781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.188066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.188340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.189050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] RBAC: Adding query filter params (name=compute) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:17.189359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:17.191252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:17.192762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:17.205456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 513/994] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:27:16 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:17.208849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.269503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 482/995] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:17 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:17.288095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf053a17-2ee8-42bc-9c3b-b24244142932 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.328790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:17.347745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04908d0c-a5f9-4021-9996-b31ed131b2ec None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:17.396707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:17.397304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.397591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.397870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.398628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:17.398939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:17.400342 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:17.405810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 514/996] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:17.415267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.472222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf053a17-2ee8-42bc-9c3b-b24244142932 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:17.530100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a696c2f-b510-417a-af1e-8ae439749318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:17.531442 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a696c2f-b510-417a-af1e-8ae439749318 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.531909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a696c2f-b510-417a-af1e-8ae439749318 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.532364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a696c2f-b510-417a-af1e-8ae439749318 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.533480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a696c2f-b510-417a-af1e-8ae439749318 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:17.536031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a696c2f-b510-417a-af1e-8ae439749318 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-6a696c2f-b510-417a-af1e-8ae439749318 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:17.556864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:17.558574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 11 14:27:17.559730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:17.566585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 483/997] 10.222.0.72 () {56 vars in 1126 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/services/object-store => generated 93 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:17.568200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034b079f-3976-4c4d-9b10-80998008addc None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.568493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034b079f-3976-4c4d-9b10-80998008addc None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.568767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034b079f-3976-4c4d-9b10-80998008addc None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.570130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 484/998] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:17.570854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034b079f-3976-4c4d-9b10-80998008addc None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.574196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9059523e-ad83-452a-b1be-da75b96b3052 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:17.600564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:17.613226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1480699a-a58d-4f64-bf86-787a9643400a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:17.614892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1480699a-a58d-4f64-bf86-787a9643400a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.615363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1480699a-a58d-4f64-bf86-787a9643400a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.615827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1480699a-a58d-4f64-bf86-787a9643400a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.617217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1480699a-a58d-4f64-bf86-787a9643400a None admin] RBAC: Adding query filter params (interface=public, service_id=b2af41348c9f45ba828e4a47fad0356d, region_id=RegionOne) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:17.617742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1480699a-a58d-4f64-bf86-787a9643400a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:17.619984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1480699a-a58d-4f64-bf86-787a9643400a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:17.626998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 515/999] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/endpoints?service_id=b2af41348c9f45ba828e4a47fad0356d&interface=public®ion_id=RegionOne => generated 194 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:17.628248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da3b99-e8b0-449b-9c69-74eb35905eb1 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.628540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da3b99-e8b0-449b-9c69-74eb35905eb1 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.628823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da3b99-e8b0-449b-9c69-74eb35905eb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.646055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f9da3b99-e8b0-449b-9c69-74eb35905eb1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:17.647194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f9da3b99-e8b0-449b-9c69-74eb35905eb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:17.655153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:17.693617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034b079f-3976-4c4d-9b10-80998008addc None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:17.724635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 516/1000] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:17 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:17.726204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9da3b99-e8b0-449b-9c69-74eb35905eb1 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.742776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:17.743914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.744202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.744531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.745703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:17.746153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:17.747858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:17.753404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 485/1001] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:17.758338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.849143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9da3b99-e8b0-449b-9c69-74eb35905eb1 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:17.858572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:17.869220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2562e36-a3bd-4123-a39f-8102a05219f1 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:17.885294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:17.885987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.886271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.886602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.887314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:17.888775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:17.905432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 517/1002] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/services => generated 1727 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:27:17.905432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:17.913776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:17.914639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:17.914929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:17.915211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] PATH_INFO: `/v3/services/image` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:17.915887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:17.916350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:17.918037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:17.932039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:17.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:17.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:17.938592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 486/1003] 10.222.0.72 () {56 vars in 1105 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/services/image => generated 86 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:17.939656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:18.053626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4962052b-5813-4ba8-b016-548765e3f110 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:18.064179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058d570b-fe5b-4af9-86e9-f87b0f5237ed None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:18.109525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:18.110643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:18.111118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:18.111578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:18.112814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:18.113337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:18.114658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:18.115724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:18.115843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:18.116173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:18.116617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:18.117805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] RBAC: Adding query filter params (name=image) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:18.118319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:18.120708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:18.129304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 518/1004] 10.222.0.72 () {56 vars in 1122 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:18.129479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 487/1005] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:17 2022] GET /identity/v3/services?name=image => generated 122 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:18.131434 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:18.136293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:18.159196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:18.242336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:18.290359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4283abba-59d7-46ac-88e8-a4af07d14a78 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:18.297892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ee2301-8f37-4e68-bb3e-fed94cebf0c0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:18.331360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f781145-802d-48aa-b8e2-1a7008d52898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:18.332034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f781145-802d-48aa-b8e2-1a7008d52898 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:18.332331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f781145-802d-48aa-b8e2-1a7008d52898 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:18.332684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f781145-802d-48aa-b8e2-1a7008d52898 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:18.334034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f781145-802d-48aa-b8e2-1a7008d52898 None admin] RBAC: Adding query filter params (type=image) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:18.334350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f781145-802d-48aa-b8e2-1a7008d52898 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:18.335882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f781145-802d-48aa-b8e2-1a7008d52898 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:18.341325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 488/1006] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:18 2022] GET /identity/v3/services?type=image => generated 122 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:18.359106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d47204-7d80-4577-a689-355b7fdec134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:18.360263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d47204-7d80-4577-a689-355b7fdec134 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:18.360833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d47204-7d80-4577-a689-355b7fdec134 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:18.361255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d47204-7d80-4577-a689-355b7fdec134 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:18.362698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d47204-7d80-4577-a689-355b7fdec134 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:18.365912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d47204-7d80-4577-a689-355b7fdec134 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:18.402459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 519/1007] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:27:18 2022] POST /identity/v3/endpoints => generated 325 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:20.339707 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30cb8b-3966-45f6-a642-c034555e648d None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:20.340561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30cb8b-3966-45f6-a642-c034555e648d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:20.340810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30cb8b-3966-45f6-a642-c034555e648d None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:20.350247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 489/1008] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:20 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:20.359344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e038f2e7-fe52-42f9-976d-d79092c628e8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:20.360116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e038f2e7-fe52-42f9-976d-d79092c628e8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:20.360454 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e038f2e7-fe52-42f9-976d-d79092c628e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:20.391303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e038f2e7-fe52-42f9-976d-d79092c628e8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:20.393374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e038f2e7-fe52-42f9-976d-d79092c628e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:20.488863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9637b6a4-76b3-42c0-afff-ae9ad3f4a5ce None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:20.489825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9637b6a4-76b3-42c0-afff-ae9ad3f4a5ce None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:20.490019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9637b6a4-76b3-42c0-afff-ae9ad3f4a5ce None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:20.491605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 490/1009] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:27:20.498104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848d516-5ca3-49e7-99bd-f8a9e2f1cf5a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:20.498734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848d516-5ca3-49e7-99bd-f8a9e2f1cf5a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:20.499467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848d516-5ca3-49e7-99bd-f8a9e2f1cf5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:20.527851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a848d516-5ca3-49e7-99bd-f8a9e2f1cf5a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:20.529985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a848d516-5ca3-49e7-99bd-f8a9e2f1cf5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:20.540484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 520/1010] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:20 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:20.564662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e038f2e7-fe52-42f9-976d-d79092c628e8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:20.641304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 491/1011] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:20 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:20.665781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848d516-5ca3-49e7-99bd-f8a9e2f1cf5a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:20.689952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:20.693171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:20.796552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e038f2e7-fe52-42f9-976d-d79092c628e8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:20.865261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848d516-5ca3-49e7-99bd-f8a9e2f1cf5a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:20.872658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:20.874034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:20.874553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:20.875040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] PATH_INFO: `/v3/services/compute` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:20.876210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:20.878777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:20.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:20.903276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 11 14:27:20.904178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:20.904933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 521/1012] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:20.914416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:20.926994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001d3527-4320-4869-8582-70950f7edd06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:20.927691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d3527-4320-4869-8582-70950f7edd06 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:20.927981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d3527-4320-4869-8582-70950f7edd06 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:20.928265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d3527-4320-4869-8582-70950f7edd06 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:20.929024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001d3527-4320-4869-8582-70950f7edd06 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:20.931633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001d3527-4320-4869-8582-70950f7edd06 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:20.970467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 492/1013] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:27:20 2022] POST /identity/v3/services => generated 242 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:27:21.112669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccf8a0d-a6f9-433a-a6de-3e8d9f1a0fad None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:21.113130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccf8a0d-a6f9-433a-a6de-3e8d9f1a0fad None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:21.113594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccf8a0d-a6f9-433a-a6de-3e8d9f1a0fad None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:21.115750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 493/1014] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:27:21.126094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e3105-5089-43b9-8f32-bcce85eade36 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:21.126635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e3105-5089-43b9-8f32-bcce85eade36 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:21.127132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e3105-5089-43b9-8f32-bcce85eade36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:21.129618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad6ba52-5e31-4305-948c-0b380adb6f7d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:21.157732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-063e3105-5089-43b9-8f32-bcce85eade36 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:21.158385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:21.159876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-063e3105-5089-43b9-8f32-bcce85eade36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:21.194985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:21.196879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:21.197329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:21.197984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:21.199230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:21.199742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:21.202067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:21.210996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 522/1015] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:27:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:21.219205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:21.298759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 494/1016] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:21 2022] POST /identity/v3/auth/tokens => generated 1803 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:21.331236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e3105-5089-43b9-8f32-bcce85eade36 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:21.335470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:21.430039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4701ee-b2b2-45ff-b3da-1edffe622d95 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:21.493468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e3105-5089-43b9-8f32-bcce85eade36 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:21.499640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:21.500723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:21.501180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:21.501615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:21.502891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] RBAC: Adding query filter params (type=compute) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:21.503377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:21.505846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:21.522247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 523/1017] 10.222.0.72 () {56 vars in 1112 bytes} [Mon Jul 11 14:27:21 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:21.524508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:21.541547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:21.542362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:21.542698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:21.542977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:21.544283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:21.545757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:21.547724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 495/1018] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:21.554139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:21.607941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:21.667877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:21.710130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba78039e-39d5-42d4-b0cc-d4c5baa06fbd None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:21.718312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d16cce3-171f-4bc3-9493-d6aa3e5eb7f8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:21.767936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d6e375-d0c2-440d-b330-068696a90d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:21.769012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d6e375-d0c2-440d-b330-068696a90d4f None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:21.769398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d6e375-d0c2-440d-b330-068696a90d4f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:21.769866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d6e375-d0c2-440d-b330-068696a90d4f None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:21.771201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d6e375-d0c2-440d-b330-068696a90d4f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:21.773780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d6e375-d0c2-440d-b330-068696a90d4f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:21.776241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6537113a-0655-493e-9908-c249fe525e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:21.777328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6537113a-0655-493e-9908-c249fe525e03 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:21.777771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6537113a-0655-493e-9908-c249fe525e03 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:21.778235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6537113a-0655-493e-9908-c249fe525e03 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:21.779589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6537113a-0655-493e-9908-c249fe525e03 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:21.781837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6537113a-0655-493e-9908-c249fe525e03 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:21.813050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 524/1019] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:27:21 2022] POST /identity/v3/endpoints => generated 334 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:27:21.818683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 496/1020] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:27:21 2022] POST /identity/v3/projects => generated 311 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:23.610303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9a418c-78b5-4273-8880-e5191da88d1e None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:23.611100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9a418c-78b5-4273-8880-e5191da88d1e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:23.611100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9a418c-78b5-4273-8880-e5191da88d1e None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:23.612511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 525/1021] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:23.619307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452935c-1b6b-476b-91b0-95361291b8cd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:23.619777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452935c-1b6b-476b-91b0-95361291b8cd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:23.620329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452935c-1b6b-476b-91b0-95361291b8cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:23.648471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1452935c-1b6b-476b-91b0-95361291b8cd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:23.650329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1452935c-1b6b-476b-91b0-95361291b8cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:23.796930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 497/1022] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:23 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:23.840287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9a418c-78b5-4273-8880-e5191da88d1e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:24.039088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:24.047838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9a418c-78b5-4273-8880-e5191da88d1e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:24.096490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b763ea1-0db1-4def-b17d-cc5e0bc1880a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:24.097307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b763ea1-0db1-4def-b17d-cc5e0bc1880a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:24.097592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b763ea1-0db1-4def-b17d-cc5e0bc1880a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:24.097875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b763ea1-0db1-4def-b17d-cc5e0bc1880a None admin] PATH_INFO: `/v3/services/image` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:24.099152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b763ea1-0db1-4def-b17d-cc5e0bc1880a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:24.100577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b763ea1-0db1-4def-b17d-cc5e0bc1880a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-8b763ea1-0db1-4def-b17d-cc5e0bc1880a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:24.115005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:24.116994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:24.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:24.127735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 526/1023] 10.222.0.72 () {56 vars in 1105 bytes} [Mon Jul 11 14:27:23 2022] GET /identity/v3/services/image => generated 86 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:24.127951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1452935c-1b6b-476b-91b0-95361291b8cd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:24.222489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:24.323043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1452935c-1b6b-476b-91b0-95361291b8cd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:24.349501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b0464c-e10b-474e-8476-08b0a433c127 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:24.350145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b0464c-e10b-474e-8476-08b0a433c127 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:24.350785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b0464c-e10b-474e-8476-08b0a433c127 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:24.353433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 527/1024] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:24.359016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484b925c-8091-4afb-a409-ab3249c77bb7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:24.359387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484b925c-8091-4afb-a409-ab3249c77bb7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:24.359558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484b925c-8091-4afb-a409-ab3249c77bb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:24.381445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-484b925c-8091-4afb-a409-ab3249c77bb7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:24.382681 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-484b925c-8091-4afb-a409-ab3249c77bb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:24.389723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:24.390910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:24.391408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:24.392227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:24.393246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] RBAC: Adding query filter params (name=image) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:24.393796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:24.397002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:24.405267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 498/1025] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:24 2022] GET /identity/v3/services?name=image => generated 122 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:24.411345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:24.502520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 528/1026] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:24 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:24.541906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484b925c-8091-4afb-a409-ab3249c77bb7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:24.562229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5336d1-55fb-4608-9705-b502a3ba6c42 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:24.584271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:24.623192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:24.624664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:24.625180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:24.625684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:24.627297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] RBAC: Adding query filter params (type=image) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:24.627870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:24.631233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:24.646947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 499/1027] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:24 2022] GET /identity/v3/services?type=image => generated 350 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:24.651682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:24.733951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:24.747850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484b925c-8091-4afb-a409-ab3249c77bb7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:24.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:24.819921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:24.820289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:24.820747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:24.822925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:24.825270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:24.827165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 529/1028] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:24.835844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:24.864437 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237be11-7e01-495e-9ba1-c49fa61fa7d3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:24.928063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:24.929799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:24.930663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:24.931385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:24.933125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] RBAC: Adding query filter params (interface=public, service_id=170d6330e30d4172a731d706da8704fc, region_id=RegionOne) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:24.933924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:24.936910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:24.947542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 500/1029] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:27:24 2022] GET /identity/v3/endpoints?service_id=170d6330e30d4172a731d706da8704fc&interface=public®ion_id=RegionOne => generated 194 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:24.953794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:24.997001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a35c5-24bd-4090-8f16-b47eaebd5c62 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:25.035214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:25.052237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615eec76-ac7c-46ac-aaef-2e4f417e35be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:25.052921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615eec76-ac7c-46ac-aaef-2e4f417e35be None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:25.053349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615eec76-ac7c-46ac-aaef-2e4f417e35be None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:25.053686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615eec76-ac7c-46ac-aaef-2e4f417e35be None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:25.054456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615eec76-ac7c-46ac-aaef-2e4f417e35be None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:25.055956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615eec76-ac7c-46ac-aaef-2e4f417e35be None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:25.096804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 530/1030] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:27:24 2022] POST /identity/v3/users => generated 285 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:25.098610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be88d04-9eea-4eb7-911e-63144d60b1eb None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:25.099264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be88d04-9eea-4eb7-911e-63144d60b1eb None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:25.099712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be88d04-9eea-4eb7-911e-63144d60b1eb None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:25.102497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 531/1031] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:25.112382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bf6da-d27a-4f5b-abeb-e556abe37519 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:25.112912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bf6da-d27a-4f5b-abeb-e556abe37519 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:25.114550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bf6da-d27a-4f5b-abeb-e556abe37519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:25.135687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-901bf6da-d27a-4f5b-abeb-e556abe37519 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:25.137204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-901bf6da-d27a-4f5b-abeb-e556abe37519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:25.157741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:25.166357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281b0901-5c31-4cda-8bcc-ee5ba76a13f3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:25.236016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:25.236919 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:25.237278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:25.237897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 532/1032] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:25 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:25.238068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:25.240755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:25.242542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:25.261242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 501/1033] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:27:24 2022] GET /identity/v3/services => generated 1957 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:27:25.280497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901bf6da-d27a-4f5b-abeb-e556abe37519 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:25.462878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901bf6da-d27a-4f5b-abeb-e556abe37519 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:25.539694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:25.541040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:25.541536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:25.542095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:25.561595 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:25.579263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:25.581052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:25.582802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 533/1034] 10.222.0.72 () {56 vars in 1120 bytes} [Mon Jul 11 14:27:25 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:25.594844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:25.755305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7d1223-7448-485b-965a-81f2fd2d1386 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:25.758197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:25.799076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:25.799774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:25.800064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:25.800378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:25.801150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:25.802005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:25.803486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:25.814331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 502/1035] 10.222.0.72 () {56 vars in 1115 bytes} [Mon Jul 11 14:27:25 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:25.822669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:25.835055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:25.945264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867aeb99-8d61-44b9-b81e-b859f0bf957a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:25.988061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:25.989405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:25.989694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:25.989970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:26.002876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:26.004372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:26.025567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:26.027506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:26.029210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:26.029210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:26.029210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:26.029210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:26.029210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 534/1036] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:26.036290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:26.163401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5136c7b7-a11b-4d0b-bd0b-8e34254ab324 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:26.205287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:26.206411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:26.206927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:26.207429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:26.208936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:26.209509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:26.212214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:26.240714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 503/1037] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:26.243079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:26.358082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:26.437710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2067dad-7641-4284-bda7-0dee9c8c4c6e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:26.504279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:26.505852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:26.506366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:26.506997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:26.523289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:26.526109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:27:26.556134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:26.557938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:26.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 535/1038] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:27:26 2022] GET /identity/v3/users/nova => generated 82 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:26.570989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:26.621287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:26.769499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad2f0f5-6b6e-4708-9c0e-30f2ea9705ab None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:26.833503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:26.834678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:26.835173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:26.835659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:26.837297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:26.837849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:26.840604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:26.878912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 504/1039] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:27:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:26.884332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:27.056136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc92a336-3cc0-47cc-9c38-510520f0d209 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:27.087713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:27.105686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8c30e5-116d-4742-9df3-de80c4761f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:27.106355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c30e5-116d-4742-9df3-de80c4761f89 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:27.106648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c30e5-116d-4742-9df3-de80c4761f89 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:27.106941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c30e5-116d-4742-9df3-de80c4761f89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:27.188453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8c30e5-116d-4742-9df3-de80c4761f89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:27.193431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8c30e5-116d-4742-9df3-de80c4761f89 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:27.322236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 536/1040] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:26 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:27.328864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:27.444968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010aa4ce-9d4e-4902-9bbd-fa1671f9a53d None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:27.445983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010aa4ce-9d4e-4902-9bbd-fa1671f9a53d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:27.446562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010aa4ce-9d4e-4902-9bbd-fa1671f9a53d None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:27.449691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 537/1041] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:27 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:27.456740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c31e664-03be-40c5-8d1d-142221c86247 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:27.457023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c31e664-03be-40c5-8d1d-142221c86247 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:27.457347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c31e664-03be-40c5-8d1d-142221c86247 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:27.480317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9c31e664-03be-40c5-8d1d-142221c86247 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:27.482395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9c31e664-03be-40c5-8d1d-142221c86247 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:27.484856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:27.506323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec8fbcd-93dc-4e59-a603-7c9202fe23df None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:27.512345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:27.551247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:27.552084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:27.552367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:27.552647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:27.567552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:27.569461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:27.594167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:27.595190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:27.596093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:27.596093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:27.596093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:27.596093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:27.596093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 505/1042] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:27.602179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:27.616399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 538/1043] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:27 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:27.640846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c31e664-03be-40c5-8d1d-142221c86247 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:27.796554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e8a2f5-da03-4c4b-8fe1-9d3ef983bae5 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:27.843870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c31e664-03be-40c5-8d1d-142221c86247 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:27.862095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:27.863097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:27.863579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:27.864130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:27.865393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:27.865926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:27.868252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:27.872184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:27.888571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 506/1044] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:27.898622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:27.906270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:27.907480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:27.907896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:27.908306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] PATH_INFO: `/v3/services/image` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:27.909111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:27.911271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:27.925941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:27.928062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 11 14:27:27.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:27.934659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 539/1045] 10.222.0.72 () {56 vars in 1105 bytes} [Mon Jul 11 14:27:27 2022] GET /identity/v3/services/image => generated 86 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:27.936598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:27.956136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:28.055640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d29bdc-c8a7-49af-8751-0f38bdc343f3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:28.096276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:28.096965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.097248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.097536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.098323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] RBAC: Adding query filter params (name=image) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:28.098675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:28.100091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:28.102914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef42f04c-d544-4ceb-9d5a-76ec300429a3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:28.105608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 540/1046] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:27 2022] GET /identity/v3/services?name=image => generated 122 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:28.111916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:28.164780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1daebb-2634-4548-b34b-49a9ce610aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:28.166168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1daebb-2634-4548-b34b-49a9ce610aea None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.166730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1daebb-2634-4548-b34b-49a9ce610aea None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.167249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1daebb-2634-4548-b34b-49a9ce610aea None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.181156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1daebb-2634-4548-b34b-49a9ce610aea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:28.182686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1daebb-2634-4548-b34b-49a9ce610aea None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-fb1daebb-2634-4548-b34b-49a9ce610aea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:27:28.203439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:28.205377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 507/1047] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92680a-6876-48e2-baaf-d92695c69ada None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.207121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92680a-6876-48e2-baaf-d92695c69ada None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.208687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92680a-6876-48e2-baaf-d92695c69ada None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.208687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 508/1048] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:28.214201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d309bd1-387c-4913-bf92-df9feb267dc2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.214516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d309bd1-387c-4913-bf92-df9feb267dc2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.214798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d309bd1-387c-4913-bf92-df9feb267dc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.241730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6d309bd1-387c-4913-bf92-df9feb267dc2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:28.242970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6d309bd1-387c-4913-bf92-df9feb267dc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:28.252935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:28.282567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57924ee3-ac16-42b5-9272-ab28d5a3d5f4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:28.320055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:28.322836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.322836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.322836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.322836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] RBAC: Adding query filter params (type=image) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:28.322836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:28.323914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:28.332530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 541/1049] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:28 2022] GET /identity/v3/services?type=image => generated 350 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:28.333423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:28.336356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:28.368590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 509/1050] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:28 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:28.370174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d309bd1-387c-4913-bf92-df9feb267dc2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:28.456986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa24c2e-a088-4822-98f0-961637d7fede None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:28.488474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d309bd1-387c-4913-bf92-df9feb267dc2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:28.510046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:28.510745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.511028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.511309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.512141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:28.512453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:28.513921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:28.536608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 542/1051] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:27:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:28.536608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:28.536908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:28.536908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.536908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.536908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.537202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:28.537202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:28.537326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:28.563903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 510/1052] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:27:28 2022] POST /identity/v3/endpoints => generated 327 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:28.563903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:28.626652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:28.737621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee816e7-02c5-4018-87e8-91c95fc1eba6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:28.755483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234eb8ab-2985-4e1a-a993-e06eda7f3eb4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:28.795530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:28.796354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.796636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.796919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.809353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:28.811753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:28.812843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:28.813335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:28.813805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:28.815447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] RBAC: Adding query filter params (role.id=3e10326cb64249db9b62814df9290687, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=2165eacee8954dce975ba81ba8ac4232) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:28.815987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:28.817236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:28.818600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:28.819478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:28.820190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:28.820264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 543/1053] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:27:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:28.824927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:28.829835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 511/1054] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:28 2022] GET /identity/v3/role_assignments?user.id=2165eacee8954dce975ba81ba8ac4232&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=3e10326cb64249db9b62814df9290687 => generated 259 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:28.903094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:28.989861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4dbd96-8dfc-4e9e-8d9f-344b475863cb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:29.056665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:29.057814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:29.058321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:29.058857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:29.060304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:29.060859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:29.063469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:29.086473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 544/1055] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:27:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:29.090201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:29.277904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5bba1c-8f9f-4286-ab68-aea4286aa89d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:29.280841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:29.322209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:29.323398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:29.323901 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:29.324327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] PATH_INFO: `/v3/users/a4863c9c26364c4cb195c5b5b3d522ee` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:29.352791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] RBAC: Authorizing `identity:get_user(user_id=a4863c9c26364c4cb195c5b5b3d522ee)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:29.354538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:29.360966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 512/1056] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:29 2022] GET /identity/v3/users/a4863c9c26364c4cb195c5b5b3d522ee => generated 285 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:29.362372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:29.533296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b136e8fd-b15d-403f-8023-69405f13472d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:29.567332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:29.584325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:29.585017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:29.585305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:29.585583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:29.687457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:29.688961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:29.837694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 545/1057] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:29 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:29.839293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:29.979036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fcbb0e-187b-4c97-82ba-4640f58d25d4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:30.029803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2671958e-1aa7-49ff-ad4c-76e867bda8ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:30.030943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2671958e-1aa7-49ff-ad4c-76e867bda8ff None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:30.031359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2671958e-1aa7-49ff-ad4c-76e867bda8ff None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:30.031820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2671958e-1aa7-49ff-ad4c-76e867bda8ff None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:30.033427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2671958e-1aa7-49ff-ad4c-76e867bda8ff None admin] RBAC: Authorizing `identity:get_project(project_id=6d1685511ab94143bc1b9df4a668f025)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:30.034978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2671958e-1aa7-49ff-ad4c-76e867bda8ff None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:30.042618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 513/1058] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:27:29 2022] GET /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025 => generated 311 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:30.046151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:30.080105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:30.181390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a666e57a-b703-452b-a3b4-022712c7ce06 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:30.225280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55f20bc-2402-4cf5-88ea-6b5245f7fad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:30.225961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f20bc-2402-4cf5-88ea-6b5245f7fad3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:30.226267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f20bc-2402-4cf5-88ea-6b5245f7fad3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:30.226685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f20bc-2402-4cf5-88ea-6b5245f7fad3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:30.227572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55f20bc-2402-4cf5-88ea-6b5245f7fad3 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=6d1685511ab94143bc1b9df4a668f025, user.id=a4863c9c26364c4cb195c5b5b3d522ee) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:30.227892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55f20bc-2402-4cf5-88ea-6b5245f7fad3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:30.229343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55f20bc-2402-4cf5-88ea-6b5245f7fad3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:30.242524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 546/1059] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:30 2022] GET /identity/v3/role_assignments?user.id=a4863c9c26364c4cb195c5b5b3d522ee&scope.project.id=6d1685511ab94143bc1b9df4a668f025&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:31.020564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48460095-d5e7-459f-95fd-07bad3e7823d None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:31.021371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48460095-d5e7-459f-95fd-07bad3e7823d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:31.021371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48460095-d5e7-459f-95fd-07bad3e7823d None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:31.023176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 514/1060] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:31.029707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21207656-d2ba-4dcd-9eda-21aa625d95b2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:31.030113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21207656-d2ba-4dcd-9eda-21aa625d95b2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:31.030458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21207656-d2ba-4dcd-9eda-21aa625d95b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:31.052434 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-21207656-d2ba-4dcd-9eda-21aa625d95b2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:31.053922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-21207656-d2ba-4dcd-9eda-21aa625d95b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:31.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 547/1061] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:31 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:31.177669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48460095-d5e7-459f-95fd-07bad3e7823d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:31.214601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:31.319858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48460095-d5e7-459f-95fd-07bad3e7823d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:31.368795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:31.369658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:31.370095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:31.370636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:31.386629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:31.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:31.398361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:31.399970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:31.402686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 515/1062] 10.222.0.72 () {56 vars in 1120 bytes} [Mon Jul 11 14:27:31 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:31.405777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21207656-d2ba-4dcd-9eda-21aa625d95b2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:31.432517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:31.527211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21207656-d2ba-4dcd-9eda-21aa625d95b2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:31.567438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:31.568334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:31.569224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:31.569224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:31.569967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:31.570208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:31.571659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:31.581741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 548/1063] 10.222.0.72 () {56 vars in 1115 bytes} [Mon Jul 11 14:27:31 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:31.587622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:31.701563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:31.723319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c975578-9970-4151-bae6-21f39bcaa30a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:31.780174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:31.781583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:31.782094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:31.782629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:31.810992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:31.812517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:31.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:31.834125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:31.835806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:31.835806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:31.835806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:31.835806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:31.838883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 516/1064] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:31.840545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:31.993884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:32.022899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35972c52-6da6-4a24-a15c-7655c7c1bf64 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:32.070899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e60a7a-7582-4bd2-a101-ed230d20fe5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:32.071761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60a7a-7582-4bd2-a101-ed230d20fe5f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.072010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60a7a-7582-4bd2-a101-ed230d20fe5f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.072410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60a7a-7582-4bd2-a101-ed230d20fe5f None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.073445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e60a7a-7582-4bd2-a101-ed230d20fe5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:32.073767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e60a7a-7582-4bd2-a101-ed230d20fe5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:32.075457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e60a7a-7582-4bd2-a101-ed230d20fe5f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:32.093603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 549/1065] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:32.103098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:32.117192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d25528-642c-406f-9b93-52e8cfe406f9 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.117613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d25528-642c-406f-9b93-52e8cfe406f9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.117893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d25528-642c-406f-9b93-52e8cfe406f9 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.120018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 550/1066] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:27:32.127346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c639f-1d02-4390-a371-3c7755563857 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.127748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c639f-1d02-4390-a371-3c7755563857 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.127905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c639f-1d02-4390-a371-3c7755563857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.151816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c12c639f-1d02-4390-a371-3c7755563857 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:32.153148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c12c639f-1d02-4390-a371-3c7755563857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:32.270263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 551/1067] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:32.300044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c639f-1d02-4390-a371-3c7755563857 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:32.305652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc673b1-ce25-48fc-b0b0-44f6dad527d0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:32.351335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:32.352172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.352506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.352835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.363306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:32.365108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:32.368490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:27:32.393146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:32.394760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:32.396282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 517/1068] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:27:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:32.401932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:32.421430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:32.470162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c639f-1d02-4390-a371-3c7755563857 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:32.516910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9b2f44-5f9c-4e2f-b182-75f2af879b35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:32.517948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9b2f44-5f9c-4e2f-b182-75f2af879b35 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.518392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9b2f44-5f9c-4e2f-b182-75f2af879b35 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.518928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9b2f44-5f9c-4e2f-b182-75f2af879b35 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.574329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de85d8d9-876e-4dbb-b77c-9d442591138e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:32.628417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9b2f44-5f9c-4e2f-b182-75f2af879b35 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:32.630038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9b2f44-5f9c-4e2f-b182-75f2af879b35 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:32.632728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285aa394-1c68-4738-812f-039b3e667ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:32.633819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285aa394-1c68-4738-812f-039b3e667ea0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.634293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285aa394-1c68-4738-812f-039b3e667ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.634796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285aa394-1c68-4738-812f-039b3e667ea0 None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.636391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285aa394-1c68-4738-812f-039b3e667ea0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:32.637110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285aa394-1c68-4738-812f-039b3e667ea0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:32.639597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285aa394-1c68-4738-812f-039b3e667ea0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:32.682054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 518/1069] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:27:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:32.684615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14434a43-4f64-4a8e-af02-340995074ef7 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.684917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14434a43-4f64-4a8e-af02-340995074ef7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.685286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14434a43-4f64-4a8e-af02-340995074ef7 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.687761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 519/1070] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:32.688859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14434a43-4f64-4a8e-af02-340995074ef7 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:32.731415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:32.774598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 552/1071] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:32 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 11 14:27:32.775741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb5f98-9f26-46a8-a6d6-1f9dd3ad4400 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.776065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb5f98-9f26-46a8-a6d6-1f9dd3ad4400 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.776350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb5f98-9f26-46a8-a6d6-1f9dd3ad4400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.778488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:32.800597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-8feb5f98-9f26-46a8-a6d6-1f9dd3ad4400 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:32.802494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8feb5f98-9f26-46a8-a6d6-1f9dd3ad4400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:32.853193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14434a43-4f64-4a8e-af02-340995074ef7 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:32.900691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 553/1072] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:32.903623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feb5f98-9f26-46a8-a6d6-1f9dd3ad4400 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:32.913058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:32.914479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:32.915000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:32.915518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:32.937119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:32.939965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:32.965545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:32.975542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:32.977605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:32.977605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:32.977605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:32.977605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:32.977605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 520/1073] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:32.977605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.046582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:33.050655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feb5f98-9f26-46a8-a6d6-1f9dd3ad4400 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.095736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.095736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.096983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.096983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.111820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:33.114371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:33.114828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e33d09-740d-4952-9b76-c0f75dc0d83f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.133906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:33.134710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:33.136280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:33.138898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:33.138898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:33.138898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:33.138898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:33.138898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 554/1074] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:33.138898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.177320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.178650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.179169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.179690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.197638 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:33.213403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:33.215238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:33.216397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 521/1075] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:27:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:33.217146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.278497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cf984b-2feb-4dfe-8f6d-4146bd1c66ca None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.333928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.335100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.335610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.336125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.337488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:33.338041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:33.340933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:33.355964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f785c74-3243-481a-b643-3c128d87d0aa None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.362620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 555/1076] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:33.364459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.390207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:33.408014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.408790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.409123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.409378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.410284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:33.410717 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:33.412529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:33.427261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 522/1077] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:33.427941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.490013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:33.549922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebf0945-0061-49e7-9dff-82c04c8419a4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.611931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.612985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.613361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.613770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.615038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:33.615719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:33.617990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:33.624126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fd822c-7bb0-4bee-b973-7409cb14fb58 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.632209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 556/1078] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:27:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:33.638419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.685181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.685457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.685916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.686204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.701430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:33.704214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:27:33.731628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:33.732763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:33.733835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:33.736151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 523/1079] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:33.745417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.766622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:33.812205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad857359-3524-4ad2-85aa-bf3d6e489877 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.862444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.863819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.864243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.864362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:33.864939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] PATH_INFO: `/v3/users/a4863c9c26364c4cb195c5b5b3d522ee` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.866468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] RBAC: Authorizing `identity:get_user(user_id=a4863c9c26364c4cb195c5b5b3d522ee)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:33.868653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:33.871330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 557/1080] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:33 2022] GET /identity/v3/users/a4863c9c26364c4cb195c5b5b3d522ee => generated 285 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:33.879053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351697ff-9b47-4c3f-9f74-ce31e564e07a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:33.881121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:33.927422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:33.928416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:33.928844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:33.929272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:33.930494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:33.930966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:33.933246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:33.960403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 524/1081] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:27:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:33.961692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:34.045766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbaa930-7d28-4ffd-b180-68a8c7a09ba6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:34.093597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:34.094665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:34.095073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:34.095460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:34.096824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] RBAC: Authorizing `identity:get_project(project_id=6d1685511ab94143bc1b9df4a668f025)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:34.097863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660ec099-ebca-46a8-bbcb-38cb3ffb08cd None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:34.098335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:34.100455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 558/1082] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:27:33 2022] GET /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025 => generated 311 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:34.107237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:34.124173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:34.143720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b0f55c-3ae0-4e1e-9018-d4bf1a2316ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:34.145777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b0f55c-3ae0-4e1e-9018-d4bf1a2316ac None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:34.146257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b0f55c-3ae0-4e1e-9018-d4bf1a2316ac None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:34.146674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b0f55c-3ae0-4e1e-9018-d4bf1a2316ac None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/2165eacee8954dce975ba81ba8ac4232/roles/3e10326cb64249db9b62814df9290687` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:34.149035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b0f55c-3ae0-4e1e-9018-d4bf1a2316ac None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=2165eacee8954dce975ba81ba8ac4232, role_id=3e10326cb64249db9b62814df9290687)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:34.150976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b0f55c-3ae0-4e1e-9018-d4bf1a2316ac None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:34.167685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 525/1083] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:27:33 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/2165eacee8954dce975ba81ba8ac4232/roles/3e10326cb64249db9b62814df9290687 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:27:34.245106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672d4bee-d0e8-48a8-8bbf-e5096cc72129 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:34.286841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805686c0-e619-44d1-980d-b6263320391a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:34.287822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805686c0-e619-44d1-980d-b6263320391a None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:34.288137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805686c0-e619-44d1-980d-b6263320391a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:34.288474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805686c0-e619-44d1-980d-b6263320391a None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025/users/a4863c9c26364c4cb195c5b5b3d522ee/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:34.291007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805686c0-e619-44d1-980d-b6263320391a None admin] RBAC: Authorizing `identity:create_grant(project_id=6d1685511ab94143bc1b9df4a668f025, user_id=a4863c9c26364c4cb195c5b5b3d522ee, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:34.292906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805686c0-e619-44d1-980d-b6263320391a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:34.311871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 559/1084] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:27:34 2022] PUT /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025/users/a4863c9c26364c4cb195c5b5b3d522ee/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:27:36.234049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4da6d80-82da-43eb-b39c-93748a1e27da None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.234647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4da6d80-82da-43eb-b39c-93748a1e27da None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.234647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4da6d80-82da-43eb-b39c-93748a1e27da None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.236978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 526/1085] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:36.244546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c84d71a-8ab3-4a45-a4f5-40316577a390 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.244990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c84d71a-8ab3-4a45-a4f5-40316577a390 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.245257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c84d71a-8ab3-4a45-a4f5-40316577a390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.264003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1c84d71a-8ab3-4a45-a4f5-40316577a390 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:36.265823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1c84d71a-8ab3-4a45-a4f5-40316577a390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:36.366457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:36.374477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 560/1086] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:36.398742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4da6d80-82da-43eb-b39c-93748a1e27da None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:36.411700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:36.467382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae51c01-8479-48c1-ac0d-179405490dc0 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.467874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae51c01-8479-48c1-ac0d-179405490dc0 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.468345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae51c01-8479-48c1-ac0d-179405490dc0 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.470837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 561/1087] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:36.475949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54efef5d-9590-47a9-a4b0-e6bf55c2e320 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.476237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54efef5d-9590-47a9-a4b0-e6bf55c2e320 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.476522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54efef5d-9590-47a9-a4b0-e6bf55c2e320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.493926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-54efef5d-9590-47a9-a4b0-e6bf55c2e320 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:36.495288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-54efef5d-9590-47a9-a4b0-e6bf55c2e320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:36.514590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4da6d80-82da-43eb-b39c-93748a1e27da None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:36.552007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:36.552824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.553103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.553380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.565020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:36.574241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:36.575578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:36.581787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 527/1088] 10.222.0.72 () {56 vars in 1120 bytes} [Mon Jul 11 14:27:36 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:36.582610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:36.582687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 562/1089] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:27:36.625772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54efef5d-9590-47a9-a4b0-e6bf55c2e320 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:36.662735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:36.701138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed051ed-183c-40a6-b051-04fb2e0ead3f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:36.744277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:36.744867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.745157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.745435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.746190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:36.746531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:36.747921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:36.750417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54efef5d-9590-47a9-a4b0-e6bf55c2e320 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:36.757380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 528/1090] 10.222.0.72 () {56 vars in 1115 bytes} [Mon Jul 11 14:27:36 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:36.763186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:36.782230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:36.793318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:36.794734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.795444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.795945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.808026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:36.815937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:36.817442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:36.818271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 563/1091] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:27:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:36.824871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:36.887798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f92e17f-a890-4bac-9af3-6375b189a0e3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:36.924990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:36.931695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:36.932500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.932782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.933055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.938674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3c63c3-57a7-4509-bf06-11c3425001f9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:36.945412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:36.946917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:36.965333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:36.966394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:36.967342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:36.967342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:36.967342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:36.967342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:36.967342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 529/1092] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:36.972266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:36.974281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:36.975196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:36.975480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:36.975764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:36.977358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:36.977673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:36.979089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:36.989201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 564/1093] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:27:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:36.994984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:37.050671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:37.075832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee60c87-bee6-4cb4-871e-59f9f8a9a8e2 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.100531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9ec725-e020-49bd-a314-73185d21a90e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.110082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.110794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.111085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.111369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.112108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:37.112422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.114350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.135713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 530/1094] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:37.136821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:37.140907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.141877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.142157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.142456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] PATH_INFO: `/v3/users/a4863c9c26364c4cb195c5b5b3d522ee` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.143696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] RBAC: Authorizing `identity:get_user(user_id=a4863c9c26364c4cb195c5b5b3d522ee)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.145685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.147391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 565/1095] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:36 2022] GET /identity/v3/users/a4863c9c26364c4cb195c5b5b3d522ee => generated 285 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:37.153812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:37.198889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:37.276033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafbb5d1-96bc-4f8c-b6f5-7010316b29f6 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.283297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694aa784-0481-4961-ae01-cefa3abb34f9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:37.317155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.317893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.318234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.318620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.336613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.337416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.337701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.337975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.347219 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.349325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:27:37.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:37.366588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:37.368852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 531/1096] 10.222.0.72 () {56 vars in 1093 bytes} [Mon Jul 11 14:27:37 2022] GET /identity/v3/users/nova => generated 82 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:27:37.372219 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:37.402523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.404030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.493739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0abf8f-3e3f-471a-a66b-4af39ac28cf9 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.496275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:37.528316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 566/1097] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:37 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 11 14:27:37.529455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:37.540356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.541435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.541901 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.542372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.543891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:37.544421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.546872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.551951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:37.567690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 532/1098] 10.222.0.72 () {56 vars in 1133 bytes} [Mon Jul 11 14:27:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:37.575775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:37.648023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42abf5cc-c9a8-4060-b197-dca6f2bf265d None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.685250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7530dff-97b8-4bac-9274-421a1a6feefc None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.686959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.687874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.688152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.688428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.689755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] RBAC: Authorizing `identity:get_project(project_id=6d1685511ab94143bc1b9df4a668f025)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.691224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.698657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 567/1099] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:27:37 2022] GET /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:37.700387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:37.735698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.736664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.737103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.737521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.781644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:37.805082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:37.821922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6c5367-54eb-4266-8203-04feefc13c2f None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:37.822501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.823965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.860585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:37.861245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:37.861533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:37.861820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:37.862690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=6d1685511ab94143bc1b9df4a668f025, user.id=a4863c9c26364c4cb195c5b5b3d522ee) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:37.863008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:37.865000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:37.871051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 568/1100] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:37 2022] GET /identity/v3/role_assignments?user.id=a4863c9c26364c4cb195c5b5b3d522ee&scope.project.id=6d1685511ab94143bc1b9df4a668f025&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:37.943189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 533/1101] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:37 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:37.949433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:38.069980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef0cc70-c629-4aac-b56b-d8310126f781 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:38.108237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:38.109088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:38.109374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:38.109653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:38.121281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:38.122781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:27:38.141666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:38.142868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:38.143966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:27:38.143966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:27:38.143966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:27:38.143966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:38.143966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 569/1102] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:38.150965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:38.239279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:38.285957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76603da9-570a-435f-9562-a8c73e6d1db4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:38.324743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:38.325581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:38.325871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:38.326155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:38.326987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:38.327305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:38.328737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:38.341115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 534/1103] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:38.347161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:38.353152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:38.461122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bde705-3d48-43b3-a720-580dc723f95c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:38.500908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:38.501706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:38.501984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:38.502322 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:38.509823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:38.511349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:27:38.526815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:38.528600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:27:38.530364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 570/1104] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:27:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:38.534704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:38.691466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dda19b-2ec9-4821-a10c-d8b2aefaa150 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:38.719177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:38.732744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70334ae-aef7-4ee1-9fc5-823e09f2d356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:38.733412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70334ae-aef7-4ee1-9fc5-823e09f2d356 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:38.733693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70334ae-aef7-4ee1-9fc5-823e09f2d356 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:38.733971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70334ae-aef7-4ee1-9fc5-823e09f2d356 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:38.734802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70334ae-aef7-4ee1-9fc5-823e09f2d356 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:38.735115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70334ae-aef7-4ee1-9fc5-823e09f2d356 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:38.736579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70334ae-aef7-4ee1-9fc5-823e09f2d356 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:38.750363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 535/1105] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:27:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:38.757452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:38.829673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:38.883157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935c805-cffa-4896-a739-6701d6291cf1 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:38.923473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91a16cf-7603-47d1-938a-b19602fb3c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:38.924249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91a16cf-7603-47d1-938a-b19602fb3c9b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:38.924541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91a16cf-7603-47d1-938a-b19602fb3c9b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:38.924915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91a16cf-7603-47d1-938a-b19602fb3c9b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:38.926260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91a16cf-7603-47d1-938a-b19602fb3c9b None admin] RBAC: Adding query filter params (role.id=3e10326cb64249db9b62814df9290687, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=2165eacee8954dce975ba81ba8ac4232) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:38.926755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91a16cf-7603-47d1-938a-b19602fb3c9b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:38.928514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91a16cf-7603-47d1-938a-b19602fb3c9b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:38.934800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 571/1106] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:38 2022] GET /identity/v3/role_assignments?user.id=2165eacee8954dce975ba81ba8ac4232&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=3e10326cb64249db9b62814df9290687 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:40.104961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2c28e-a2e6-4ad8-83c8-f24a94b9f241 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:40.105737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2c28e-a2e6-4ad8-83c8-f24a94b9f241 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:40.105737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2c28e-a2e6-4ad8-83c8-f24a94b9f241 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:40.106957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 536/1107] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:40.112526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d8d68-922e-45b7-8c94-4fd87addc5e8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:40.113118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d8d68-922e-45b7-8c94-4fd87addc5e8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:40.113118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d8d68-922e-45b7-8c94-4fd87addc5e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:40.130685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-236d8d68-922e-45b7-8c94-4fd87addc5e8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:40.131960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-236d8d68-922e-45b7-8c94-4fd87addc5e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:40.206070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 572/1108] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:40.232181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c2c28e-a2e6-4ad8-83c8-f24a94b9f241 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:40.340694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c2c28e-a2e6-4ad8-83c8-f24a94b9f241 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:40.385251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b583c15-d919-45d0-827b-f57938f9eb50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:40.385949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b583c15-d919-45d0-827b-f57938f9eb50 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:40.386240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b583c15-d919-45d0-827b-f57938f9eb50 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:40.386542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b583c15-d919-45d0-827b-f57938f9eb50 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:40.387773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b583c15-d919-45d0-827b-f57938f9eb50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:40.389240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b583c15-d919-45d0-827b-f57938f9eb50 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:40.390930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 537/1109] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:40.396930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d8d68-922e-45b7-8c94-4fd87addc5e8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:40.458191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:40.513068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d8d68-922e-45b7-8c94-4fd87addc5e8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:40.550230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfcc2df-ab98-4774-9b64-e9f82a7d2b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:40.550990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfcc2df-ab98-4774-9b64-e9f82a7d2b25 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:40.551305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfcc2df-ab98-4774-9b64-e9f82a7d2b25 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:40.551675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfcc2df-ab98-4774-9b64-e9f82a7d2b25 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:40.552592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfcc2df-ab98-4774-9b64-e9f82a7d2b25 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:40.554084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfcc2df-ab98-4774-9b64-e9f82a7d2b25 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:40.577046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 573/1110] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:27:40 2022] POST /identity/v3/users => generated 286 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:42.678483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceee3025-fcef-40bb-8369-5eb03c85208e None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:42.679022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceee3025-fcef-40bb-8369-5eb03c85208e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:42.679158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceee3025-fcef-40bb-8369-5eb03c85208e None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:42.680452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 538/1111] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:42.686588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0fae3c-2e7b-4448-a103-fb21c16bcc55 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:42.687109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0fae3c-2e7b-4448-a103-fb21c16bcc55 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:42.687244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0fae3c-2e7b-4448-a103-fb21c16bcc55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:42.705989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-af0fae3c-2e7b-4448-a103-fb21c16bcc55 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:42.707337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-af0fae3c-2e7b-4448-a103-fb21c16bcc55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:42.783756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 574/1112] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:42 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:42.807676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceee3025-fcef-40bb-8369-5eb03c85208e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:42.864633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:42.921013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceee3025-fcef-40bb-8369-5eb03c85208e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:42.960854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:42.961635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:42.961921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:42.962200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] PATH_INFO: `/v3/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:42.963732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] RBAC: Authorizing `identity:get_role(role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:42.965142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:42.966574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 539/1113] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:42 2022] GET /identity/v3/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 223 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:42.976083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0fae3c-2e7b-4448-a103-fb21c16bcc55 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:43.028237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:43.089581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0fae3c-2e7b-4448-a103-fb21c16bcc55 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:43.125272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:43.126169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:43.126482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:43.126761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] PATH_INFO: `/v3/users/38cb73d721a44a9babdde3269eaf8b3e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:43.141694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] RBAC: Authorizing `identity:get_user(user_id=38cb73d721a44a9babdde3269eaf8b3e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:43.143241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:43.145057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 575/1114] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:42 2022] GET /identity/v3/users/38cb73d721a44a9babdde3269eaf8b3e => generated 286 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:43.151184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:43.262598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5709cd-27c4-46d0-8ea2-bc046f0ec571 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:43.285067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:43.304846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:43.305517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:43.305800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:43.306324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:43.382213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:43.383721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:43.492015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 540/1115] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:43 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:43.498324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:43.629551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11b08e-0308-421d-ad39-5a260f5efef8 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:43.631425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:43.667996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41750c-6300-4e5d-9f94-01afb2d9584c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:43.668373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41750c-6300-4e5d-9f94-01afb2d9584c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:43.671388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41750c-6300-4e5d-9f94-01afb2d9584c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:43.671388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41750c-6300-4e5d-9f94-01afb2d9584c None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:43.671388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41750c-6300-4e5d-9f94-01afb2d9584c None admin] RBAC: Authorizing `identity:get_project(project_id=6d1685511ab94143bc1b9df4a668f025)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:43.671751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41750c-6300-4e5d-9f94-01afb2d9584c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:43.673536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 576/1116] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:27:43 2022] GET /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025 => generated 311 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:43.684658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:43.709904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:43.806728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eda279-a8a9-4e5d-88fd-46e8a536af5d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:43.862015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b90929f-ac25-429f-a264-d03ed4749ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:43.863060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90929f-ac25-429f-a264-d03ed4749ce9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:43.864050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90929f-ac25-429f-a264-d03ed4749ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:43.864504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90929f-ac25-429f-a264-d03ed4749ce9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:43.865932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b90929f-ac25-429f-a264-d03ed4749ce9 None admin] RBAC: Adding query filter params (role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=6d1685511ab94143bc1b9df4a668f025, user.id=38cb73d721a44a9babdde3269eaf8b3e) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:43.866471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b90929f-ac25-429f-a264-d03ed4749ce9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:43.869006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b90929f-ac25-429f-a264-d03ed4749ce9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:43.879268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 541/1117] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:43 2022] GET /identity/v3/role_assignments?user.id=38cb73d721a44a9babdde3269eaf8b3e&scope.project.id=6d1685511ab94143bc1b9df4a668f025&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:45.913198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951104f0-8049-4806-bdbf-a7331e74bb5d None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:45.913753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951104f0-8049-4806-bdbf-a7331e74bb5d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:45.913892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951104f0-8049-4806-bdbf-a7331e74bb5d None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:45.915809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 577/1118] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:45.921421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b5ef17-752e-4546-bcdc-d65e4d1a25f1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:45.921855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b5ef17-752e-4546-bcdc-d65e4d1a25f1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:45.922351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b5ef17-752e-4546-bcdc-d65e4d1a25f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:45.940045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-03b5ef17-752e-4546-bcdc-d65e4d1a25f1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:45.941152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-03b5ef17-752e-4546-bcdc-d65e4d1a25f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:46.009749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 542/1119] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:45 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:46.033317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951104f0-8049-4806-bdbf-a7331e74bb5d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:46.149094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951104f0-8049-4806-bdbf-a7331e74bb5d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:46.184614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:46.185477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:46.185764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:46.186039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] PATH_INFO: `/v3/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:46.187551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] RBAC: Authorizing `identity:get_role(role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:46.188969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:46.190421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 578/1120] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:46 2022] GET /identity/v3/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 223 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:46.197131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b5ef17-752e-4546-bcdc-d65e4d1a25f1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:46.212267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:46.303788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b5ef17-752e-4546-bcdc-d65e4d1a25f1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:46.341146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:46.342009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:46.342291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:46.342748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] PATH_INFO: `/v3/users/38cb73d721a44a9babdde3269eaf8b3e` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:46.343950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] RBAC: Authorizing `identity:get_user(user_id=38cb73d721a44a9babdde3269eaf8b3e)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:46.345409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:46.347137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 543/1121] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:46 2022] GET /identity/v3/users/38cb73d721a44a9babdde3269eaf8b3e => generated 286 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:46.353859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:46.405785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:46.475149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d015130-ac61-4fea-a199-dd89a81de4f3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:46.512472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f38b1-889d-4f5c-ae26-be182cd7f4c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:46.513464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f38b1-889d-4f5c-ae26-be182cd7f4c2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:46.513855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f38b1-889d-4f5c-ae26-be182cd7f4c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:46.514234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f38b1-889d-4f5c-ae26-be182cd7f4c2 None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:46.515779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62f38b1-889d-4f5c-ae26-be182cd7f4c2 None admin] RBAC: Authorizing `identity:get_project(project_id=6d1685511ab94143bc1b9df4a668f025)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:46.517253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62f38b1-889d-4f5c-ae26-be182cd7f4c2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:46.519050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 579/1122] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:27:46 2022] GET /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025 => generated 311 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:46.525307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:46.610611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:46.631574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698fc118-8cab-4726-8477-5277432cede6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:46.666648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f1881e-144f-4c9a-ba5a-9ed026ed8586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:46.667550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f1881e-144f-4c9a-ba5a-9ed026ed8586 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:46.667835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f1881e-144f-4c9a-ba5a-9ed026ed8586 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:46.668114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f1881e-144f-4c9a-ba5a-9ed026ed8586 None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025/users/38cb73d721a44a9babdde3269eaf8b3e/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:46.670389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f1881e-144f-4c9a-ba5a-9ed026ed8586 None admin] RBAC: Authorizing `identity:create_grant(project_id=6d1685511ab94143bc1b9df4a668f025, user_id=38cb73d721a44a9babdde3269eaf8b3e, role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:46.671898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f1881e-144f-4c9a-ba5a-9ed026ed8586 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:46.686285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 544/1123] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:27:46 2022] PUT /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025/users/38cb73d721a44a9babdde3269eaf8b3e/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:27:48.732750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150b8a23-f722-4e07-bf36-4bed3a84e673 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:48.733280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150b8a23-f722-4e07-bf36-4bed3a84e673 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:48.733372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150b8a23-f722-4e07-bf36-4bed3a84e673 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:48.734726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 580/1124] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:48.740311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab1ecd-0a28-4286-8a32-1a688bbf6d01 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:48.740799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab1ecd-0a28-4286-8a32-1a688bbf6d01 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:48.740891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab1ecd-0a28-4286-8a32-1a688bbf6d01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:48.759573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9aab1ecd-0a28-4286-8a32-1a688bbf6d01 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:48.760836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9aab1ecd-0a28-4286-8a32-1a688bbf6d01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:48.834942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 545/1125] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:48 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:48.858326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150b8a23-f722-4e07-bf36-4bed3a84e673 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:48.964812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150b8a23-f722-4e07-bf36-4bed3a84e673 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:48.973172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:49.006291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:49.007198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:49.007486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:49.007764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] PATH_INFO: `/v3/roles/e418c49286d246b5ac35fc27b6cafa5a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:49.009274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] RBAC: Authorizing `identity:get_role(role_id=e418c49286d246b5ac35fc27b6cafa5a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:49.010691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:49.012084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 581/1126] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:48 2022] GET /identity/v3/roles/e418c49286d246b5ac35fc27b6cafa5a => generated 223 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:49.018128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab1ecd-0a28-4286-8a32-1a688bbf6d01 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:49.089914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:49.123789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab1ecd-0a28-4286-8a32-1a688bbf6d01 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:49.158236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:49.159140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:49.159450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:49.159726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] PATH_INFO: `/v3/users/38cb73d721a44a9babdde3269eaf8b3e` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:49.161001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] RBAC: Authorizing `identity:get_user(user_id=38cb73d721a44a9babdde3269eaf8b3e)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:49.162464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:49.164125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 546/1127] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:49 2022] GET /identity/v3/users/38cb73d721a44a9babdde3269eaf8b3e => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:49.170195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:49.281330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f899a630-2e12-4efe-9600-98c2622fd14a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:49.317780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:49.318457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:49.318753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:49.319035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:49.367620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:49.394143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:49.395707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:49.505929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 582/1128] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:49 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:49.511987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:49.624384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73541be7-262b-4189-b238-9613a05956e6 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:49.661710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88466013-1496-4445-8b07-ffad61289edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:49.662722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88466013-1496-4445-8b07-ffad61289edf None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:49.663012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88466013-1496-4445-8b07-ffad61289edf None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:49.663289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88466013-1496-4445-8b07-ffad61289edf None admin] PATH_INFO: `/v3/projects/6d1685511ab94143bc1b9df4a668f025` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:49.664610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88466013-1496-4445-8b07-ffad61289edf None admin] RBAC: Authorizing `identity:get_project(project_id=6d1685511ab94143bc1b9df4a668f025)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:49.666173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88466013-1496-4445-8b07-ffad61289edf None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:49.668078 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 547/1129] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:27:49 2022] GET /identity/v3/projects/6d1685511ab94143bc1b9df4a668f025 => generated 311 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:27:49.674808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:49.758137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:49.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a451f7a-6d09-477c-bdb0-378e2b409e81 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:49.825435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5bf761-c736-42b8-be1e-dd9ddaa969bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:49.826104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5bf761-c736-42b8-be1e-dd9ddaa969bf None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:49.826384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5bf761-c736-42b8-be1e-dd9ddaa969bf None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:49.826689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5bf761-c736-42b8-be1e-dd9ddaa969bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:49.827547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5bf761-c736-42b8-be1e-dd9ddaa969bf None admin] RBAC: Adding query filter params (role.id=e418c49286d246b5ac35fc27b6cafa5a, scope.project.id=6d1685511ab94143bc1b9df4a668f025, user.id=38cb73d721a44a9babdde3269eaf8b3e) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:49.827860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5bf761-c736-42b8-be1e-dd9ddaa969bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:49.829594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5bf761-c736-42b8-be1e-dd9ddaa969bf None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:49.836447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 583/1130] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:49 2022] GET /identity/v3/role_assignments?user.id=38cb73d721a44a9babdde3269eaf8b3e&scope.project.id=6d1685511ab94143bc1b9df4a668f025&role.id=e418c49286d246b5ac35fc27b6cafa5a => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:51.944787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0a313-abbd-4f79-81fd-b4ca63dd6e53 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:51.945522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0a313-abbd-4f79-81fd-b4ca63dd6e53 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:51.945522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0a313-abbd-4f79-81fd-b4ca63dd6e53 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:51.946697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 548/1131] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:51.952915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4abf0b-c7ae-415a-804d-b04d745bdf82 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:51.953551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4abf0b-c7ae-415a-804d-b04d745bdf82 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:51.953551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4abf0b-c7ae-415a-804d-b04d745bdf82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:51.971730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-cd4abf0b-c7ae-415a-804d-b04d745bdf82 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:51.973559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cd4abf0b-c7ae-415a-804d-b04d745bdf82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:52.050528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 584/1132] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:51 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:52.075162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef0a313-abbd-4f79-81fd-b4ca63dd6e53 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:52.085073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:52.191100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef0a313-abbd-4f79-81fd-b4ca63dd6e53 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:52.234058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c18be4-e187-4762-85e2-419c9aaf9cae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:52.234936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c18be4-e187-4762-85e2-419c9aaf9cae None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:52.235228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c18be4-e187-4762-85e2-419c9aaf9cae None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:52.235509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c18be4-e187-4762-85e2-419c9aaf9cae None admin] PATH_INFO: `/v3/domains/default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:52.236726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c18be4-e187-4762-85e2-419c9aaf9cae None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:52.238712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c18be4-e187-4762-85e2-419c9aaf9cae None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:52.240387 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 549/1133] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:52.246630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4abf0b-c7ae-415a-804d-b04d745bdf82 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:52.322761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:52.357651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4abf0b-c7ae-415a-804d-b04d745bdf82 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:52.397993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a386e-840e-4163-990b-cc49e7660b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:52.398700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a386e-840e-4163-990b-cc49e7660b3c None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:52.399025 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a386e-840e-4163-990b-cc49e7660b3c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:52.399264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a386e-840e-4163-990b-cc49e7660b3c None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:52.399994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a386e-840e-4163-990b-cc49e7660b3c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:52.401479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a386e-840e-4163-990b-cc49e7660b3c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:52.430188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 585/1134] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:27:52 2022] POST /identity/v3/projects => generated 311 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:54.577368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59509c-c664-4d78-b741-31a3d9a1e246 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:54.577938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59509c-c664-4d78-b741-31a3d9a1e246 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:54.578073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59509c-c664-4d78-b741-31a3d9a1e246 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:54.579774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 550/1135] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:54.586450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39071888-725a-4a4d-b03c-e16817594160 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:54.586998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39071888-725a-4a4d-b03c-e16817594160 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:54.586998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39071888-725a-4a4d-b03c-e16817594160 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:54.607146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-39071888-725a-4a4d-b03c-e16817594160 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:54.608409 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-39071888-725a-4a4d-b03c-e16817594160 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:54.688233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 586/1136] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:54 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:54.714446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59509c-c664-4d78-b741-31a3d9a1e246 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:54.821211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:54.852524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59509c-c664-4d78-b741-31a3d9a1e246 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:54.889873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1320a7-bb17-427f-9af1-1090ec59db31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:54.890719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1320a7-bb17-427f-9af1-1090ec59db31 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:54.891003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1320a7-bb17-427f-9af1-1090ec59db31 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:54.891360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1320a7-bb17-427f-9af1-1090ec59db31 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:54.892617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1320a7-bb17-427f-9af1-1090ec59db31 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:54.894088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1320a7-bb17-427f-9af1-1090ec59db31 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:54.895834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 551/1137] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:27:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:54.901946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39071888-725a-4a4d-b03c-e16817594160 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:54.961212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:55.030685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39071888-725a-4a4d-b03c-e16817594160 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:55.081210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cfdc99-c0ed-4805-81df-82f2acb5c068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:55.082413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cfdc99-c0ed-4805-81df-82f2acb5c068 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:55.082825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cfdc99-c0ed-4805-81df-82f2acb5c068 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:55.083018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cfdc99-c0ed-4805-81df-82f2acb5c068 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:55.083766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cfdc99-c0ed-4805-81df-82f2acb5c068 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:55.085202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cfdc99-c0ed-4805-81df-82f2acb5c068 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:55.107349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 587/1138] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:27:54 2022] POST /identity/v3/users => generated 286 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:27:57.311346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0757e-d6b9-4efa-a68d-75fcc5b35bb8 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:57.311909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0757e-d6b9-4efa-a68d-75fcc5b35bb8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:57.311909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0757e-d6b9-4efa-a68d-75fcc5b35bb8 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:57.313276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 552/1139] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:27:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:27:57.318753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4871f3-153c-4c89-98f4-0d28cedaa8d8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:57.319281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4871f3-153c-4c89-98f4-0d28cedaa8d8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:57.319421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4871f3-153c-4c89-98f4-0d28cedaa8d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:57.337059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-eb4871f3-153c-4c89-98f4-0d28cedaa8d8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:27:57.338296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-eb4871f3-153c-4c89-98f4-0d28cedaa8d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:27:57.410415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 588/1140] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:27:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:27:57.434257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0757e-d6b9-4efa-a68d-75fcc5b35bb8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:57.542069 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0757e-d6b9-4efa-a68d-75fcc5b35bb8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:57.576391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:57.577136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:57.577417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:57.577693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:57.588645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:27:57.596227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:27:57.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:27:57.598577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 553/1141] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:27:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:27:57.603954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4871f3-153c-4c89-98f4-0d28cedaa8d8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:57.640691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:57.708185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4871f3-153c-4c89-98f4-0d28cedaa8d8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:57.744137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:57.744809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:57.745095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:57.745378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:57.746128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:57.746464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:57.747858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:57.756935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 589/1142] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:27:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:57.763543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:57.767076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:57.888981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2baf24-623e-4643-83da-74cda7fdc351 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:57.923680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:57.924554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:57.924836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:57.925120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] PATH_INFO: `/v3/users/d6a1fe80c47245109937a9f0389f3d45` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:57.940619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] RBAC: Authorizing `identity:get_user(user_id=d6a1fe80c47245109937a9f0389f3d45)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:57.942210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:57.943993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 554/1143] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:27:57 2022] GET /identity/v3/users/d6a1fe80c47245109937a9f0389f3d45 => generated 286 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:57.949869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:58.050156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:58.054972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a9b3b5-b5fb-40b1-a21a-4206cfaf6c19 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:58.090815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:58.091471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:58.091755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:58.092035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:58.161304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:58.162834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:58.259238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:58.263918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 590/1144] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:27:57 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:27:58.270845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:58.329708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:27:58.371410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111e48d-4f69-4fc7-90d9-900a4afed84e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:58.407360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bc61fc-fded-45be-8c60-a221a7b319e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:58.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc61fc-fded-45be-8c60-a221a7b319e9 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:58.408516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc61fc-fded-45be-8c60-a221a7b319e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:58.408793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc61fc-fded-45be-8c60-a221a7b319e9 None admin] PATH_INFO: `/v3/projects/774507edd8b248f0b40cbc710387148d` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:58.410005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bc61fc-fded-45be-8c60-a221a7b319e9 None admin] RBAC: Authorizing `identity:get_project(project_id=774507edd8b248f0b40cbc710387148d)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:58.411666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bc61fc-fded-45be-8c60-a221a7b319e9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:58.413323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 555/1145] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:27:58 2022] GET /identity/v3/projects/774507edd8b248f0b40cbc710387148d => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:27:58.419511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:27:58.539929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e92134-2066-4f84-aca5-a37d4329cf4b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:27:58.578405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ac696f-e1e7-47a6-8ce4-5a7e79a7e7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:27:58.579275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ac696f-e1e7-47a6-8ce4-5a7e79a7e7f9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:27:58.579552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ac696f-e1e7-47a6-8ce4-5a7e79a7e7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:27:58.579828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ac696f-e1e7-47a6-8ce4-5a7e79a7e7f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:27:58.580658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ac696f-e1e7-47a6-8ce4-5a7e79a7e7f9 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=774507edd8b248f0b40cbc710387148d, user.id=d6a1fe80c47245109937a9f0389f3d45) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:27:58.580970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ac696f-e1e7-47a6-8ce4-5a7e79a7e7f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:27:58.582479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ac696f-e1e7-47a6-8ce4-5a7e79a7e7f9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:27:58.588089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 591/1146] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:27:58 2022] GET /identity/v3/role_assignments?user.id=d6a1fe80c47245109937a9f0389f3d45&scope.project.id=774507edd8b248f0b40cbc710387148d&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:00.601869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5afa073-c099-4aa0-8241-10648e6908bd None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:00.602410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5afa073-c099-4aa0-8241-10648e6908bd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:00.602410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5afa073-c099-4aa0-8241-10648e6908bd None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:00.604746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 556/1147] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:00.610383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80da06a-b246-45c4-a0bb-4081fc9c0e54 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:00.610897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80da06a-b246-45c4-a0bb-4081fc9c0e54 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:00.610988 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80da06a-b246-45c4-a0bb-4081fc9c0e54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:00.628296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b80da06a-b246-45c4-a0bb-4081fc9c0e54 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:00.630004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b80da06a-b246-45c4-a0bb-4081fc9c0e54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:00.689619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:00.702126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 592/1148] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:00 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:00.725454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5afa073-c099-4aa0-8241-10648e6908bd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:00.830214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5afa073-c099-4aa0-8241-10648e6908bd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:00.858028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:00.866955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:00.867762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:00.868043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:00.868318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:00.880101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:28:00.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:28:00.889654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:28:00.895923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 557/1149] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:28:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:28:00.896756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80da06a-b246-45c4-a0bb-4081fc9c0e54 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:00.996646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80da06a-b246-45c4-a0bb-4081fc9c0e54 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:01.030952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:01.031617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:01.031899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:01.032181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:01.032929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:01.033236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:01.034630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:01.043365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 593/1150] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:28:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:01.050715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:01.152357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2376c9f-5d43-4c6c-b6c6-ad121563f48e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:01.185774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:01.186690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:01.186982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:01.187261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] PATH_INFO: `/v3/users/d6a1fe80c47245109937a9f0389f3d45` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:01.188476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] RBAC: Authorizing `identity:get_user(user_id=d6a1fe80c47245109937a9f0389f3d45)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:01.189933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:01.191632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 558/1151] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:28:01 2022] GET /identity/v3/users/d6a1fe80c47245109937a9f0389f3d45 => generated 286 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:01.197714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:01.255041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:01.301465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf880c33-3243-4950-ab3d-775ad4164461 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:01.336143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c6290c-03f1-4cf7-ac9b-c2512e622820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:01.337012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c6290c-03f1-4cf7-ac9b-c2512e622820 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:01.337297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c6290c-03f1-4cf7-ac9b-c2512e622820 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:01.337573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c6290c-03f1-4cf7-ac9b-c2512e622820 None admin] PATH_INFO: `/v3/projects/774507edd8b248f0b40cbc710387148d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:01.338776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c6290c-03f1-4cf7-ac9b-c2512e622820 None admin] RBAC: Authorizing `identity:get_project(project_id=774507edd8b248f0b40cbc710387148d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:01.340269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c6290c-03f1-4cf7-ac9b-c2512e622820 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:01.341997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 594/1152] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:28:01 2022] GET /identity/v3/projects/774507edd8b248f0b40cbc710387148d => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:01.349178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:01.406589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:01.451498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd60ed6f-4cf8-489e-9ce1-c9468f40ca23 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:01.485097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133482e9-c908-49b9-8f14-f5330c155ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:01.485983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133482e9-c908-49b9-8f14-f5330c155ef9 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:01.486266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133482e9-c908-49b9-8f14-f5330c155ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:01.486675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133482e9-c908-49b9-8f14-f5330c155ef9 None admin] PATH_INFO: `/v3/projects/774507edd8b248f0b40cbc710387148d/users/d6a1fe80c47245109937a9f0389f3d45/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:01.488674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133482e9-c908-49b9-8f14-f5330c155ef9 None admin] RBAC: Authorizing `identity:create_grant(project_id=774507edd8b248f0b40cbc710387148d, user_id=d6a1fe80c47245109937a9f0389f3d45, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:01.490111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133482e9-c908-49b9-8f14-f5330c155ef9 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:01.504296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 559/1153] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:28:01 2022] PUT /identity/v3/projects/774507edd8b248f0b40cbc710387148d/users/d6a1fe80c47245109937a9f0389f3d45/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:28:03.504438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2c9c4-a476-469b-845b-8e9227a990a5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:03.505031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2c9c4-a476-469b-845b-8e9227a990a5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:03.505031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2c9c4-a476-469b-845b-8e9227a990a5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:03.506505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 595/1154] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:03.512477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdb2439-d877-43b0-87a1-ba2322d5a6c2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:03.513046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdb2439-d877-43b0-87a1-ba2322d5a6c2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:03.513046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdb2439-d877-43b0-87a1-ba2322d5a6c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:03.532864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fbdb2439-d877-43b0-87a1-ba2322d5a6c2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:03.534536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fbdb2439-d877-43b0-87a1-ba2322d5a6c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:03.602937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 560/1155] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:03.626589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2c9c4-a476-469b-845b-8e9227a990a5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:03.730659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2c9c4-a476-469b-845b-8e9227a990a5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:03.753037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:03.765495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:03.766221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:03.766530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:03.766917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:03.777836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:28:03.785500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:28:03.786743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:28:03.787870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 596/1156] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:28:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:28:03.792935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdb2439-d877-43b0-87a1-ba2322d5a6c2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:03.843385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:03.897136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdb2439-d877-43b0-87a1-ba2322d5a6c2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:03.931461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:03.932130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:03.932417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:03.932695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:03.933444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:03.933754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:03.935167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:03.943260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 561/1157] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:28:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:03.949032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:04.049635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5d41fa-8a30-4538-b90a-0a99dfc510a4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:04.083448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:04.084316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:04.084601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:04.084878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] PATH_INFO: `/v3/users/d6a1fe80c47245109937a9f0389f3d45` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:04.086059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] RBAC: Authorizing `identity:get_user(user_id=d6a1fe80c47245109937a9f0389f3d45)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:04.087555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:04.089210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 597/1158] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:28:03 2022] GET /identity/v3/users/d6a1fe80c47245109937a9f0389f3d45 => generated 286 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:04.095261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:04.197251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5daace-d176-450b-8620-1fc18956d19a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:04.206986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:04.231784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:04.232441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:04.232730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:04.233013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:04.299967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:04.301377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:04.402903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 562/1159] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:28:04 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:28:04.408908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:04.458780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:04.508002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d494e97b-3b60-43c6-aeb9-2287233d0fbb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:04.540762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc9504e-21df-4d66-94b2-23528eb6b1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:04.541619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9504e-21df-4d66-94b2-23528eb6b1f3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:04.541901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9504e-21df-4d66-94b2-23528eb6b1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:04.542174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9504e-21df-4d66-94b2-23528eb6b1f3 None admin] PATH_INFO: `/v3/projects/774507edd8b248f0b40cbc710387148d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:04.543368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc9504e-21df-4d66-94b2-23528eb6b1f3 None admin] RBAC: Authorizing `identity:get_project(project_id=774507edd8b248f0b40cbc710387148d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:04.544802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc9504e-21df-4d66-94b2-23528eb6b1f3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:04.546621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 598/1160] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:28:04 2022] GET /identity/v3/projects/774507edd8b248f0b40cbc710387148d => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:04.552797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:04.562980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:04.653127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36dd9bd-c079-4ee4-b038-69a1c1090cab None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:04.690524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16c3bc2-74e0-4cd0-88b6-f9a6a522b948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:04.691208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c3bc2-74e0-4cd0-88b6-f9a6a522b948 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:04.691496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c3bc2-74e0-4cd0-88b6-f9a6a522b948 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:04.691779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c3bc2-74e0-4cd0-88b6-f9a6a522b948 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:04.692613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c3bc2-74e0-4cd0-88b6-f9a6a522b948 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=774507edd8b248f0b40cbc710387148d, user.id=d6a1fe80c47245109937a9f0389f3d45) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:04.692935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c3bc2-74e0-4cd0-88b6-f9a6a522b948 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:04.694399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c3bc2-74e0-4cd0-88b6-f9a6a522b948 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:04.700535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 563/1161] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:28:04 2022] GET /identity/v3/role_assignments?user.id=d6a1fe80c47245109937a9f0389f3d45&scope.project.id=774507edd8b248f0b40cbc710387148d&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:06.770574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d958a09-9aad-4635-971f-9bdcb01c5dfc None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:06.771139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d958a09-9aad-4635-971f-9bdcb01c5dfc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:06.771139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d958a09-9aad-4635-971f-9bdcb01c5dfc None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:06.772523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 599/1162] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:06.778604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9122b8-6b7f-4a4b-a6ec-0ed35784f4b9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:06.779080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9122b8-6b7f-4a4b-a6ec-0ed35784f4b9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:06.779290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9122b8-6b7f-4a4b-a6ec-0ed35784f4b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:06.801888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-dd9122b8-6b7f-4a4b-a6ec-0ed35784f4b9 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:06.803495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dd9122b8-6b7f-4a4b-a6ec-0ed35784f4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:06.873635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 564/1163] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:06.898057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d958a09-9aad-4635-971f-9bdcb01c5dfc None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:07.010864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d958a09-9aad-4635-971f-9bdcb01c5dfc None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:07.030369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:07.049668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:07.050350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:07.050875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:07.051043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:07.126722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:07.128251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:07.241589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 600/1164] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:28:06 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:28:07.247965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9122b8-6b7f-4a4b-a6ec-0ed35784f4b9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:07.250388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:07.359834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9122b8-6b7f-4a4b-a6ec-0ed35784f4b9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:07.397434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69a4b3-02e5-4bfc-b29f-3932e205409d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:07.398299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69a4b3-02e5-4bfc-b29f-3932e205409d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:07.398624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69a4b3-02e5-4bfc-b29f-3932e205409d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:07.398939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69a4b3-02e5-4bfc-b29f-3932e205409d None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:07.406345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69a4b3-02e5-4bfc-b29f-3932e205409d None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:07.407903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69a4b3-02e5-4bfc-b29f-3932e205409d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5d69a4b3-02e5-4bfc-b29f-3932e205409d None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:28:07.422623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:28:07.424246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:28:07.425758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 565/1165] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:28:07 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:28:07.430833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:07.450587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:07.535083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368285bb-8533-464f-9159-fd70b6d2f6d3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:07.570328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f89d38-1a69-48d1-b678-b2c1247bc21d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:07.571021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f89d38-1a69-48d1-b678-b2c1247bc21d None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:07.571305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f89d38-1a69-48d1-b678-b2c1247bc21d None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:07.571580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f89d38-1a69-48d1-b678-b2c1247bc21d None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:07.572298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f89d38-1a69-48d1-b678-b2c1247bc21d None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:07.572615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f89d38-1a69-48d1-b678-b2c1247bc21d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:07.573987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f89d38-1a69-48d1-b678-b2c1247bc21d None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:07.582630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 601/1166] 10.222.0.72 () {56 vars in 1115 bytes} [Mon Jul 11 14:28:07 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:09.637060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42433ff7-a75e-47cd-a8e7-5865788a9b96 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:09.637597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42433ff7-a75e-47cd-a8e7-5865788a9b96 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:09.637597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42433ff7-a75e-47cd-a8e7-5865788a9b96 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:09.639044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 566/1167] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:09.645353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f5808f-3e76-42ca-8ea4-13fda5c69a73 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:09.645793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f5808f-3e76-42ca-8ea4-13fda5c69a73 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:09.645899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f5808f-3e76-42ca-8ea4-13fda5c69a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:09.662714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-23f5808f-3e76-42ca-8ea4-13fda5c69a73 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:09.663937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-23f5808f-3e76-42ca-8ea4-13fda5c69a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:09.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 602/1168] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:09 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:09.757310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42433ff7-a75e-47cd-a8e7-5865788a9b96 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:09.822981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:09.863291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42433ff7-a75e-47cd-a8e7-5865788a9b96 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:09.896883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e555d0ae-6d78-45f8-8445-682c4f0c59a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:09.897459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e555d0ae-6d78-45f8-8445-682c4f0c59a3 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:09.897742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e555d0ae-6d78-45f8-8445-682c4f0c59a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:09.898019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e555d0ae-6d78-45f8-8445-682c4f0c59a3 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:09.898795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e555d0ae-6d78-45f8-8445-682c4f0c59a3 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:09.900158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e555d0ae-6d78-45f8-8445-682c4f0c59a3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:09.918292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 567/1169] 10.222.0.72 () {60 vars in 1138 bytes} [Mon Jul 11 14:28:09 2022] POST /identity/v3/domains => generated 267 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:28:11.996979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041829ae-5696-4f7f-81c0-058882d0ed40 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:11.997518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041829ae-5696-4f7f-81c0-058882d0ed40 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:11.997622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041829ae-5696-4f7f-81c0-058882d0ed40 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:11.999040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 603/1170] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:12.004747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a499783-7d85-43e8-8cf7-f8369632e9ac None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:12.005168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a499783-7d85-43e8-8cf7-f8369632e9ac None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:12.005320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a499783-7d85-43e8-8cf7-f8369632e9ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:12.022701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2a499783-7d85-43e8-8cf7-f8369632e9ac None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:12.023908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2a499783-7d85-43e8-8cf7-f8369632e9ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:12.097825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 568/1171] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:12.120968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041829ae-5696-4f7f-81c0-058882d0ed40 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:12.136103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:12.221210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041829ae-5696-4f7f-81c0-058882d0ed40 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:12.256300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9bc538-93e4-4493-8913-4e6c58d0f3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:12.257079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9bc538-93e4-4493-8913-4e6c58d0f3a3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:12.257363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9bc538-93e4-4493-8913-4e6c58d0f3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:12.257639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9bc538-93e4-4493-8913-4e6c58d0f3a3 None admin] PATH_INFO: `/v3/domains/7d7f62f6eea5448f8ea871556d7dee7c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:12.267948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9bc538-93e4-4493-8913-4e6c58d0f3a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=7d7f62f6eea5448f8ea871556d7dee7c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:12.269923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9bc538-93e4-4493-8913-4e6c58d0f3a3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:12.271596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 604/1172] 10.222.0.72 () {56 vars in 1183 bytes} [Mon Jul 11 14:28:12 2022] GET /identity/v3/domains/7d7f62f6eea5448f8ea871556d7dee7c => generated 267 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:12.277528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a499783-7d85-43e8-8cf7-f8369632e9ac None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:12.330069 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:12.378496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a499783-7d85-43e8-8cf7-f8369632e9ac None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:12.413033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a05a52-694b-4fd5-8774-d45f264cb880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:12.413699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a05a52-694b-4fd5-8774-d45f264cb880 None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:12.413986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a05a52-694b-4fd5-8774-d45f264cb880 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:12.414267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a05a52-694b-4fd5-8774-d45f264cb880 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:12.415022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a05a52-694b-4fd5-8774-d45f264cb880 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:12.416463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a05a52-694b-4fd5-8774-d45f264cb880 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:12.441338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 569/1173] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:28:12 2022] POST /identity/v3/projects => generated 361 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:28:14.543160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c21f721-ef59-478e-a411-70a611204aa5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:14.543697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c21f721-ef59-478e-a411-70a611204aa5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:14.543697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c21f721-ef59-478e-a411-70a611204aa5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:14.545089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 605/1174] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:14.550792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da23fcdf-5093-4649-b628-ff93332624f0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:14.551259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da23fcdf-5093-4649-b628-ff93332624f0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:14.551363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da23fcdf-5093-4649-b628-ff93332624f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:14.568474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-da23fcdf-5093-4649-b628-ff93332624f0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:14.569661 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-da23fcdf-5093-4649-b628-ff93332624f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:14.642972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 570/1175] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:14 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:14.676877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c21f721-ef59-478e-a411-70a611204aa5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:14.755185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:14.786929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c21f721-ef59-478e-a411-70a611204aa5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:14.820412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f92414-86fd-4de2-9f7f-9f6d9aecd01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:14.821198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f92414-86fd-4de2-9f7f-9f6d9aecd01c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:14.821482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f92414-86fd-4de2-9f7f-9f6d9aecd01c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:14.821757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f92414-86fd-4de2-9f7f-9f6d9aecd01c None admin] PATH_INFO: `/v3/domains/7d7f62f6eea5448f8ea871556d7dee7c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:14.822966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f92414-86fd-4de2-9f7f-9f6d9aecd01c None admin] RBAC: Authorizing `identity:get_domain(domain_id=7d7f62f6eea5448f8ea871556d7dee7c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:14.824414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f92414-86fd-4de2-9f7f-9f6d9aecd01c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:14.826310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 606/1176] 10.222.0.72 () {56 vars in 1183 bytes} [Mon Jul 11 14:28:14 2022] GET /identity/v3/domains/7d7f62f6eea5448f8ea871556d7dee7c => generated 267 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:14.832321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da23fcdf-5093-4649-b628-ff93332624f0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:14.866038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:14.931989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da23fcdf-5093-4649-b628-ff93332624f0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:14.965526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7733daf-4849-45e9-87bf-2c6216d184ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:14.966341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7733daf-4849-45e9-87bf-2c6216d184ed None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:14.966509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7733daf-4849-45e9-87bf-2c6216d184ed None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:14.966787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7733daf-4849-45e9-87bf-2c6216d184ed None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:14.967513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7733daf-4849-45e9-87bf-2c6216d184ed None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:14.968957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7733daf-4849-45e9-87bf-2c6216d184ed None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:14.990259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 571/1177] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:28:14 2022] POST /identity/v3/users => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:28:17.054574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634f576-85cf-4d89-9f8a-2515de83b804 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:17.055168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634f576-85cf-4d89-9f8a-2515de83b804 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:17.055168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634f576-85cf-4d89-9f8a-2515de83b804 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:17.056494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 607/1178] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:17.062024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4a70d-b24c-4729-bc60-1bb94ca4c7fd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:17.062479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4a70d-b24c-4729-bc60-1bb94ca4c7fd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:17.062697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4a70d-b24c-4729-bc60-1bb94ca4c7fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:17.079896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f2d4a70d-b24c-4729-bc60-1bb94ca4c7fd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:17.081079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f2d4a70d-b24c-4729-bc60-1bb94ca4c7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:17.145925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 572/1179] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:17 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:17.170360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634f576-85cf-4d89-9f8a-2515de83b804 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:17.272204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634f576-85cf-4d89-9f8a-2515de83b804 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:17.307731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:17.308735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:17.309004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:17.309318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:17.317361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:17.320714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:28:17.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:28:17.330090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:17.330980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:28:17.331937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 608/1180] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:28:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:28:17.336526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d4a70d-b24c-4729-bc60-1bb94ca4c7fd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:17.359235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:17.442907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d4a70d-b24c-4729-bc60-1bb94ca4c7fd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:17.478616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:17.479295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:17.479575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:17.479851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:17.480600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:17.480915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:17.482293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:17.495456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 573/1181] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:28:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:17.496815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:17.597775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0aa418-2996-490e-8d92-0ea35b5d9370 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:17.629910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:17.630817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:17.631105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:17.631382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] PATH_INFO: `/v3/users/a07333fec4fe4fc89352d5dd3d1adf4c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:17.644999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] RBAC: Authorizing `identity:get_user(user_id=a07333fec4fe4fc89352d5dd3d1adf4c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:17.646478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:17.648146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 609/1182] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:28:17 2022] GET /identity/v3/users/a07333fec4fe4fc89352d5dd3d1adf4c => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:17.654072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:17.735645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:17.754489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d70e0-1103-44ba-abdf-4c8634ad0abb None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:17.789685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:17.790331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:17.790647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:17.790922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:17.857799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:17.859239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:17.936628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:17.955253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 574/1183] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:28:17 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:28:17.961277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:18.020526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:18.066615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbe68a1-ba95-4f29-8019-0663582d0dd3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:18.103483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e773d1aa-7218-4945-be4b-a367a414c490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:18.104350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e773d1aa-7218-4945-be4b-a367a414c490 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:18.104636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e773d1aa-7218-4945-be4b-a367a414c490 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:18.104915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e773d1aa-7218-4945-be4b-a367a414c490 None admin] PATH_INFO: `/v3/projects/040cc11c06ea4b78b787d0afc391f175` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:18.106159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e773d1aa-7218-4945-be4b-a367a414c490 None admin] RBAC: Authorizing `identity:get_project(project_id=040cc11c06ea4b78b787d0afc391f175)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:18.107667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e773d1aa-7218-4945-be4b-a367a414c490 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:18.109511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 610/1184] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:28:17 2022] GET /identity/v3/projects/040cc11c06ea4b78b787d0afc391f175 => generated 361 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:18.115981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:18.221953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd03da01-dd1c-4d53-b9a0-740002b694cb None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:18.259013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62165e2-212d-4a18-aeb8-cc5042cc5c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:18.259693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62165e2-212d-4a18-aeb8-cc5042cc5c60 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:18.259981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62165e2-212d-4a18-aeb8-cc5042cc5c60 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:18.260264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62165e2-212d-4a18-aeb8-cc5042cc5c60 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:18.261109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62165e2-212d-4a18-aeb8-cc5042cc5c60 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=040cc11c06ea4b78b787d0afc391f175, user.id=a07333fec4fe4fc89352d5dd3d1adf4c) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:18.261429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62165e2-212d-4a18-aeb8-cc5042cc5c60 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:18.262907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62165e2-212d-4a18-aeb8-cc5042cc5c60 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:18.269093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 575/1185] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:28:18 2022] GET /identity/v3/role_assignments?user.id=a07333fec4fe4fc89352d5dd3d1adf4c&scope.project.id=040cc11c06ea4b78b787d0afc391f175&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:20.270233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65cd866-4d79-46ec-82de-3108cbe6bd26 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:20.270843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65cd866-4d79-46ec-82de-3108cbe6bd26 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:20.270843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65cd866-4d79-46ec-82de-3108cbe6bd26 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:20.272181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 611/1186] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:20.277614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b611695-2b93-4297-97bf-f7ec1a6e6688 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:20.278092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b611695-2b93-4297-97bf-f7ec1a6e6688 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:20.278198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b611695-2b93-4297-97bf-f7ec1a6e6688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:20.295731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2b611695-2b93-4297-97bf-f7ec1a6e6688 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:20.296922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2b611695-2b93-4297-97bf-f7ec1a6e6688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:20.341012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:20.380603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 576/1187] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:20 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:28:20.403263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65cd866-4d79-46ec-82de-3108cbe6bd26 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:20.506899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65cd866-4d79-46ec-82de-3108cbe6bd26 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:20.533423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:20.543090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:20.543885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:20.544197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:20.544581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:20.554942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:28:20.563026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:28:20.564307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:28:20.565601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 612/1188] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:28:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:28:20.570633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b611695-2b93-4297-97bf-f7ec1a6e6688 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:20.676894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b611695-2b93-4297-97bf-f7ec1a6e6688 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:20.712352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:20.713028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:20.713314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:20.713592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:20.714345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:20.714686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:20.716229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:20.727115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 577/1189] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:28:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:20.733239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:20.836545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d95593-1c0e-45bd-9e72-72f001f88d64 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:20.869778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:20.870689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:20.870973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:20.871250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] PATH_INFO: `/v3/users/a07333fec4fe4fc89352d5dd3d1adf4c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:20.872448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] RBAC: Authorizing `identity:get_user(user_id=a07333fec4fe4fc89352d5dd3d1adf4c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:20.874455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:20.876111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 613/1190] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:28:20 2022] GET /identity/v3/users/a07333fec4fe4fc89352d5dd3d1adf4c => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:20.882758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:20.923308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:20.983236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae75c5e-674b-4fb7-83bb-dbe25e945fd5 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:21.016176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7b8f06-fa7b-4451-8a5b-091514a2a0d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:21.017040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b8f06-fa7b-4451-8a5b-091514a2a0d6 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:21.017323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b8f06-fa7b-4451-8a5b-091514a2a0d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:21.017597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b8f06-fa7b-4451-8a5b-091514a2a0d6 None admin] PATH_INFO: `/v3/projects/040cc11c06ea4b78b787d0afc391f175` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:21.019333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7b8f06-fa7b-4451-8a5b-091514a2a0d6 None admin] RBAC: Authorizing `identity:get_project(project_id=040cc11c06ea4b78b787d0afc391f175)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:21.020783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7b8f06-fa7b-4451-8a5b-091514a2a0d6 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:21.022494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 578/1191] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:28:20 2022] GET /identity/v3/projects/040cc11c06ea4b78b787d0afc391f175 => generated 361 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:21.029693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:21.086823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:21.133164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edca826-2360-405f-be8a-a999ab635678 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:21.166323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ec21d4-d9cb-4233-b291-c0ae9ab25145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:21.167241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec21d4-d9cb-4233-b291-c0ae9ab25145 None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:21.167520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec21d4-d9cb-4233-b291-c0ae9ab25145 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:21.167789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec21d4-d9cb-4233-b291-c0ae9ab25145 None admin] PATH_INFO: `/v3/projects/040cc11c06ea4b78b787d0afc391f175/users/a07333fec4fe4fc89352d5dd3d1adf4c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:21.169777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ec21d4-d9cb-4233-b291-c0ae9ab25145 None admin] RBAC: Authorizing `identity:create_grant(project_id=040cc11c06ea4b78b787d0afc391f175, user_id=a07333fec4fe4fc89352d5dd3d1adf4c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:21.171415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ec21d4-d9cb-4233-b291-c0ae9ab25145 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:21.185349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 614/1192] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:28:21 2022] PUT /identity/v3/projects/040cc11c06ea4b78b787d0afc391f175/users/a07333fec4fe4fc89352d5dd3d1adf4c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:28:23.207460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca96a36f-87f7-4da6-a5c4-22d0b370823a None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:23.208022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca96a36f-87f7-4da6-a5c4-22d0b370823a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:23.208022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca96a36f-87f7-4da6-a5c4-22d0b370823a None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:23.209368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 579/1193] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:28:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:28:23.214641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d32d76f-4de1-4d3f-b8bd-e503273b304b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:23.215072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d32d76f-4de1-4d3f-b8bd-e503273b304b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:23.215214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d32d76f-4de1-4d3f-b8bd-e503273b304b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:23.232248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2d32d76f-4de1-4d3f-b8bd-e503273b304b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:28:23.233438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2d32d76f-4de1-4d3f-b8bd-e503273b304b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:28:23.306016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 615/1194] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:28:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:28:23.330518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca96a36f-87f7-4da6-a5c4-22d0b370823a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:23.431641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca96a36f-87f7-4da6-a5c4-22d0b370823a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:23.438836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:23.466925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:23.467639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:23.467931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:23.468216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:23.478268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:28:23.485940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:28:23.487625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:28:23.488608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 580/1195] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:28:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:28:23.493736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d32d76f-4de1-4d3f-b8bd-e503273b304b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:23.536946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:23.590485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d32d76f-4de1-4d3f-b8bd-e503273b304b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:23.623206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:23.623863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:23.624142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:23.624417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:23.625158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:23.625462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:23.626886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:23.635735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 616/1196] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:28:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:23.641828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:23.741221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbfc2d-183d-4c72-adbf-a114a584e0d4 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:23.774011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:23.774905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:23.775188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:23.775464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] PATH_INFO: `/v3/users/a07333fec4fe4fc89352d5dd3d1adf4c` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:23.776632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] RBAC: Authorizing `identity:get_user(user_id=a07333fec4fe4fc89352d5dd3d1adf4c)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:23.778067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:23.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 581/1197] 10.222.0.72 () {56 vars in 1177 bytes} [Mon Jul 11 14:28:23 2022] GET /identity/v3/users/a07333fec4fe4fc89352d5dd3d1adf4c => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:23.786317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:23.887293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d11f8-1c07-4c53-858c-353be4b57a5a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:23.894196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:23.921480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:23.922137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:23.922424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:23.922731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:23.989570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:23.991067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:24.089046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 617/1198] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:28:23 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:28:24.094880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:24.129850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:24.195573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076015b2-9fa6-49e4-a03b-4bf0efbe6e1a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:24.229280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189d39e8-771a-44d4-bd91-6aa1af41370b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:24.230135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189d39e8-771a-44d4-bd91-6aa1af41370b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:24.230423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189d39e8-771a-44d4-bd91-6aa1af41370b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:24.230726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189d39e8-771a-44d4-bd91-6aa1af41370b None admin] PATH_INFO: `/v3/projects/040cc11c06ea4b78b787d0afc391f175` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:24.231971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189d39e8-771a-44d4-bd91-6aa1af41370b None admin] RBAC: Authorizing `identity:get_project(project_id=040cc11c06ea4b78b787d0afc391f175)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:24.233397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189d39e8-771a-44d4-bd91-6aa1af41370b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:24.235097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 582/1199] 10.222.0.72 () {56 vars in 1186 bytes} [Mon Jul 11 14:28:24 2022] GET /identity/v3/projects/040cc11c06ea4b78b787d0afc391f175 => generated 361 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:24.241292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:28:24.249474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:24.340555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9dc49e-7859-448b-ad20-a196d384d684 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:28:24.373892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7853298-123c-4ca5-ab45-875a83bbdbd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:28:24.374586 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7853298-123c-4ca5-ab45-875a83bbdbd3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:28:24.374875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7853298-123c-4ca5-ab45-875a83bbdbd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:28:24.375160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7853298-123c-4ca5-ab45-875a83bbdbd3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:28:24.375992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7853298-123c-4ca5-ab45-875a83bbdbd3 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=040cc11c06ea4b78b787d0afc391f175, user.id=a07333fec4fe4fc89352d5dd3d1adf4c) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:28:24.376313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7853298-123c-4ca5-ab45-875a83bbdbd3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:28:24.377764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7853298-123c-4ca5-ab45-875a83bbdbd3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:28:24.383675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 618/1200] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:28:24 2022] GET /identity/v3/role_assignments?user.id=a07333fec4fe4fc89352d5dd3d1adf4c&scope.project.id=040cc11c06ea4b78b787d0afc391f175&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:28:34.225136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=49 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:28:34.381052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=65 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:24.341887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618916f5-f596-447c-af58-4a8112733603 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:24.342911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618916f5-f596-447c-af58-4a8112733603 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:24.342911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618916f5-f596-447c-af58-4a8112733603 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:24.345385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 583/1201] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:29:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:29:24.352838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f40ae-87f5-4465-89ad-34059865ea67 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:24.353715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f40ae-87f5-4465-89ad-34059865ea67 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:24.353887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f40ae-87f5-4465-89ad-34059865ea67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:24.382689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4c2f40ae-87f5-4465-89ad-34059865ea67 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:29:24.384933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4c2f40ae-87f5-4465-89ad-34059865ea67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:29:24.507330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 619/1202] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:29:24 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:29:24.539580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618916f5-f596-447c-af58-4a8112733603 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:24.745189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618916f5-f596-447c-af58-4a8112733603 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:24.800556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:24.801410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:24.801703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:24.801979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:24.815555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:24.817128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:24.851388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:24.853122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:24.855135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:24.855135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:24.855135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:24.855135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:24.855135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 584/1203] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:24.865648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f40ae-87f5-4465-89ad-34059865ea67 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:25.107125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f40ae-87f5-4465-89ad-34059865ea67 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:25.164708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:25.175005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b36db98-2428-4896-810b-9edfde5ffd26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:25.176152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36db98-2428-4896-810b-9edfde5ffd26 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:25.176634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36db98-2428-4896-810b-9edfde5ffd26 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:25.177054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36db98-2428-4896-810b-9edfde5ffd26 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:25.178184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36db98-2428-4896-810b-9edfde5ffd26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:25.178749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36db98-2428-4896-810b-9edfde5ffd26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:25.180434 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36db98-2428-4896-810b-9edfde5ffd26 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:25.202188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 620/1204] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:25.215588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:25.348596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:25.453064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dfac1d-5668-477b-b8b8-c5da39f1b891 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:25.521419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29985a63-20e1-4ae7-ac85-7591a4e6575a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:25.522939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29985a63-20e1-4ae7-ac85-7591a4e6575a None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:25.523538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29985a63-20e1-4ae7-ac85-7591a4e6575a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:25.523970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29985a63-20e1-4ae7-ac85-7591a4e6575a None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:25.525096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29985a63-20e1-4ae7-ac85-7591a4e6575a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:25.527212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29985a63-20e1-4ae7-ac85-7591a4e6575a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:25.582470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 585/1205] 10.222.0.72 () {60 vars in 1132 bytes} [Mon Jul 11 14:29:25 2022] POST /identity/v3/users => generated 251 bytes in 360 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:29:28.736468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ec79d7-8863-4995-be7b-bb11c8ff53ec None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:28.737332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ec79d7-8863-4995-be7b-bb11c8ff53ec None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:28.737635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ec79d7-8863-4995-be7b-bb11c8ff53ec None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:28.741942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 621/1206] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:29:28 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:29:28.751252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e072c-f28c-475f-a995-78d938e2e7ed None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:28.751947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e072c-f28c-475f-a995-78d938e2e7ed None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:28.752222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e072c-f28c-475f-a995-78d938e2e7ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:28.783090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2a2e072c-f28c-475f-a995-78d938e2e7ed None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:29:28.785579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2a2e072c-f28c-475f-a995-78d938e2e7ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:29:28.927425 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 586/1207] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:29:28 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:29:28.958380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ec79d7-8863-4995-be7b-bb11c8ff53ec None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:29.193410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ec79d7-8863-4995-be7b-bb11c8ff53ec None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:29.267789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:29.269178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:29.269622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:29.270090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:29.289809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:29:29.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:29.311978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:29.314467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 622/1208] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:29:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:29.320427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2e072c-f28c-475f-a995-78d938e2e7ed None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:29.395287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:29.509982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2e072c-f28c-475f-a995-78d938e2e7ed None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:29.585065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:29.586580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:29.587059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:29.587753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:29.588924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:29.589475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:29.592069 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:29.608511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 587/1209] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:29:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:29.619437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:29.752577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:29.862740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f9fb8b-40c5-488e-8724-dc6fd3fe32f1 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:29.925583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:29.927536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:29.928271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:29.929013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:29.951074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:29.954777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:29.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:29.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:29.999725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:29.999725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:29.999725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:29.999725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:29.999725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 623/1210] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:30.008137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:30.210114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:30.214308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96378710-44e2-4199-bb09-23f88d10d583 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:30.270454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:30.271878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:30.272519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:30.273559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:30.275439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:30.275439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:30.278864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:30.299338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 588/1211] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:30.310784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:30.508257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ac8c4a-e947-4628-84f8-6d9f4fbbd877 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:30.512510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:30.584739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:30.585995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:30.586297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:30.586632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:30.597152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:30.598650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:29:30.623776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:30.635668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:30.638127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 624/1212] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:30 2022] GET /identity/v3/users/placement => generated 87 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:30.644234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:30.821285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ff2ece-5da2-4a8e-8de0-cb85d1b95b2f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:30.893024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:30.894285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:30.895135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:30.895701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:30.897664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:30.898276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:30.901866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:30.939145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 589/1213] 10.222.0.72 () {56 vars in 1143 bytes} [Mon Jul 11 14:29:30 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:30.947639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:31.122797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf8fd1-e557-4f1f-b071-a617bf7d43c9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:31.181278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:31.182420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:31.182861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:31.183165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:31.232930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:31.302812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:31.304764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:31.466242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 625/1214] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:29:30 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 519 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:29:31.478466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:31.523786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:31.674840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e2ad24-8c24-4e96-b3be-663cf8720dee None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:31.727152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:31.728621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:31.729205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:31.729752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:31.751014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:31.753407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:31.787412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:31.788985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:31.790233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:31.790233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:31.790233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:31.790233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:31.790233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 590/1215] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:31.801113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:31.925414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:32.014051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a768bc26-4829-4291-9687-9ffc85f71d37 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:32.088511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:32.089735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:32.090206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:32.090728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:32.092117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:32.093489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:32.095101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:32.117761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 626/1216] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:32.126355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:32.256796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:32.318337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb93fabc-2617-433a-a1c7-933b429bd89b None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:32.398269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:32.400636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:32.400636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:32.401361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:32.415945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:32.418354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:29:32.442028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:32.444311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:32.445810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 591/1217] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:29:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:32.455364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:32.706826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3cbfc8-4f05-4982-9ab1-0d94fbc6203b None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:32.759894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:32.791165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33287113-6793-44f3-8790-ce210dd17973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:32.792091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33287113-6793-44f3-8790-ce210dd17973 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:32.792449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33287113-6793-44f3-8790-ce210dd17973 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:32.792810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33287113-6793-44f3-8790-ce210dd17973 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:32.793777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33287113-6793-44f3-8790-ce210dd17973 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:32.794168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33287113-6793-44f3-8790-ce210dd17973 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:32.796242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33287113-6793-44f3-8790-ce210dd17973 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:32.815582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 627/1218] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:29:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:32.828043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:33.029679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdff1b71-673d-487b-8685-62edd3308a48 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:33.078110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:33.088216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2a838d-fddb-413b-b5dc-1c600e4481d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:33.089601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a838d-fddb-413b-b5dc-1c600e4481d1 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:33.090085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a838d-fddb-413b-b5dc-1c600e4481d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:33.090596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a838d-fddb-413b-b5dc-1c600e4481d1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:33.092431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2a838d-fddb-413b-b5dc-1c600e4481d1 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=5aabd82e0fd04345b63c346660e0da54) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:33.092870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2a838d-fddb-413b-b5dc-1c600e4481d1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:33.096597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2a838d-fddb-413b-b5dc-1c600e4481d1 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:33.118486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 592/1219] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:29:32 2022] GET /identity/v3/role_assignments?user.id=5aabd82e0fd04345b63c346660e0da54&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 259 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:35.921414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a04e51-074a-45f1-b5a5-5b3be0733dde None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:35.922479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a04e51-074a-45f1-b5a5-5b3be0733dde None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:35.922479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a04e51-074a-45f1-b5a5-5b3be0733dde None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:35.924912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 628/1220] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:29:35 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:29:35.933058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0631ee1-bdf8-460a-9f82-0713dd7d3814 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:35.933594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0631ee1-bdf8-460a-9f82-0713dd7d3814 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:35.933839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0631ee1-bdf8-460a-9f82-0713dd7d3814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:35.962960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a0631ee1-bdf8-460a-9f82-0713dd7d3814 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:29:35.964998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a0631ee1-bdf8-460a-9f82-0713dd7d3814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:29:36.073645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 593/1221] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:29:35 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:29:36.103108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a04e51-074a-45f1-b5a5-5b3be0733dde None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:36.284667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a04e51-074a-45f1-b5a5-5b3be0733dde None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:36.360952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:36.362567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:36.363076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:36.363679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:36.387497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:29:36.404472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:36.405634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:29:36.406690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:36.407423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 629/1222] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:29:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:36.417591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0631ee1-bdf8-460a-9f82-0713dd7d3814 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:36.605989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0631ee1-bdf8-460a-9f82-0713dd7d3814 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:36.652092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:36.672212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:36.673922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:36.673922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:36.674417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:36.675855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] RBAC: Adding query filter params (name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:36.676401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:36.679075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:36.693882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 594/1223] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:29:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:36.699913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:36.760581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:36.857217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91417290-0c23-4887-a1fd-c8b2762b6fc5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:36.915193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:36.916380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:36.916876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:36.917341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:36.935470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:36.938661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:36.968463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:36.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:36.972607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:36.972607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:36.972607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:36.972607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:36.972607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 630/1224] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:36.981488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:37.168471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1527bdd0-1760-46e9-be98-122743f290de None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:37.215932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:37.216772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:37.217260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:37.217622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:37.218768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:37.219321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:37.221875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:37.239540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 595/1225] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:37.245587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:37.409291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:37.419761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2c5ff0-bfce-4190-b462-ebf49dcb931c None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:37.477560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:37.478639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:37.478999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:37.479347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] PATH_INFO: `/v3/users/placement` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:37.491358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:37.492996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:29:37.510599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:37.512632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:37.514827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 631/1226] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:37 2022] GET /identity/v3/users/placement => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:37.520870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:37.604074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:37.708582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4bd368-7afd-4fe0-b23d-94d419b096c0 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:37.770307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:37.771040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:37.771382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:37.771647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:37.772456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:37.772761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:37.774250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:37.800887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 596/1227] 10.222.0.72 () {56 vars in 1143 bytes} [Mon Jul 11 14:29:37 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:37.808820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:37.973840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7c36e1-a8b6-4713-b3b2-6879157375df None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:38.030969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:38.032442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:38.032950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:38.033446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:38.054163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:38.056643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:38.083924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:38.085924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:38.088286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:38.088286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:38.088286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:38.088286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:38.088286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 632/1228] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:38.096210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:38.252134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e57940c-559d-4d0f-aa54-bfd9e116df7e None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:38.258143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:38.340866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:38.342467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:38.343388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:38.343951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:38.345893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:38.346806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:38.350125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:38.381747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 597/1229] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:38.389707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:38.403945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:38.540473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff787c81-c078-4619-af4e-b0ad6865cbb9 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:38.600592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:38.601908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:38.602247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:38.602695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:38.615377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:38.617600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:29:38.643038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:38.646925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:38.648612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 633/1230] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:29:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:38.656828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:38.822570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbc3812-dad6-4343-810c-75c1028c6c83 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:38.869151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda2c383-6135-487e-b268-a88d55113fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:38.870092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda2c383-6135-487e-b268-a88d55113fd8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:38.870890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda2c383-6135-487e-b268-a88d55113fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:38.871358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda2c383-6135-487e-b268-a88d55113fd8 None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:38.872792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda2c383-6135-487e-b268-a88d55113fd8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:38.873116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda2c383-6135-487e-b268-a88d55113fd8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:38.874640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda2c383-6135-487e-b268-a88d55113fd8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:38.893706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 598/1231] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:29:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:38.904429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:38.995979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:39.054564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a9af35-4f79-48f5-9013-810dad94f812 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:39.102385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eb4456-52f6-4caa-aeb2-b1cb8b78d8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:39.103940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eb4456-52f6-4caa-aeb2-b1cb8b78d8cd None admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:39.104445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eb4456-52f6-4caa-aeb2-b1cb8b78d8cd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:39.104948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eb4456-52f6-4caa-aeb2-b1cb8b78d8cd None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/5aabd82e0fd04345b63c346660e0da54/roles/b032af1c9f7a447a98b75130f5bab37a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:39.131991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eb4456-52f6-4caa-aeb2-b1cb8b78d8cd None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=5aabd82e0fd04345b63c346660e0da54, role_id=b032af1c9f7a447a98b75130f5bab37a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:39.134505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eb4456-52f6-4caa-aeb2-b1cb8b78d8cd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:39.154413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 634/1232] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:29:38 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/5aabd82e0fd04345b63c346660e0da54/roles/b032af1c9f7a447a98b75130f5bab37a => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:29:42.175603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cf21b4-b38d-450c-9c8b-058a1c06591e None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:42.176529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cf21b4-b38d-450c-9c8b-058a1c06591e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:42.176679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cf21b4-b38d-450c-9c8b-058a1c06591e None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:42.179227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 599/1233] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:29:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:29:42.188692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907660a4-ef7d-4365-8cc5-0e529f12fb97 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:42.189529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907660a4-ef7d-4365-8cc5-0e529f12fb97 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:42.189712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907660a4-ef7d-4365-8cc5-0e529f12fb97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:42.221532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-907660a4-ef7d-4365-8cc5-0e529f12fb97 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:29:42.223026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-907660a4-ef7d-4365-8cc5-0e529f12fb97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:29:42.317201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 635/1234] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:29:42 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:29:42.350122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cf21b4-b38d-450c-9c8b-058a1c06591e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:42.397150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:42.536083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cf21b4-b38d-450c-9c8b-058a1c06591e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:42.600660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:42.601950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:42.602316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:42.602721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:42.626938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:29:42.644144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:42.645309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 11 14:29:42.646488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:42.647451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 600/1235] 10.222.0.72 () {56 vars in 1102 bytes} [Mon Jul 11 14:29:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:42.661540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907660a4-ef7d-4365-8cc5-0e529f12fb97 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:42.728235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:42.835997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907660a4-ef7d-4365-8cc5-0e529f12fb97 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:42.910376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:42.912002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:42.912596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:42.913465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:42.915255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] RBAC: Adding query filter params (name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:42.915999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:42.919613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:42.937926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 636/1236] 10.222.0.72 () {56 vars in 1103 bytes} [Mon Jul 11 14:29:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:42.947865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:43.097479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:43.130012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca50a7-73b6-4a10-9d37-0ee147e79735 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:43.171117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:43.171921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:43.172750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:43.173043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:43.188513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:43.191256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:43.216118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:43.218399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:43.220436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:43.220436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:43.220436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:43.220436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:43.220436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 601/1237] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:43.225360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:43.409949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90096997-cebf-48f4-9942-f4e527562624 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:43.430257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:43.484264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:43.485392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:43.485850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:43.486338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:43.487703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:43.488255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:43.491424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:43.508746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 637/1238] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:43.515431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:43.641587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53444eb3-3aa5-47bb-be8e-3b4b41b36249 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:43.700188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:43.701337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:43.701725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:43.702036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] PATH_INFO: `/v3/users/placement` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:43.706722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:43.715564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:43.718332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:29:43.748535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:43.749846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:43.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:43.751720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 602/1239] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:43 2022] GET /identity/v3/users/placement => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:43.757880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:43.939282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8c13fc-6efa-4658-bb7b-c1937d917b4a None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:44.018086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:44.019300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:44.019796 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:44.020263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:44.021920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:44.022517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:44.025444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:44.061035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 638/1240] 10.222.0.72 () {56 vars in 1143 bytes} [Mon Jul 11 14:29:43 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:44.069102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:44.264586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5102efbb-c945-4234-a3fa-e8b24bc582de None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:44.336414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:44.337586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:44.338093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:44.338595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:44.469645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:44.479869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:44.482164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:44.634189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 603/1241] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:29:44 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 566 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:29:44.643635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:44.683970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:44.808233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9f365-79d7-4cde-9449-fb6a2b9c1dfd None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:44.877189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:44.877979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:44.878343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:44.878825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:44.896518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:44.899192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:44.931592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:44.933098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:44.934516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:44.934516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:44.934516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:44.934516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:44.934516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 639/1242] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:44.940670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:45.104687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:45.115334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a12ee6-e5d6-412e-a12c-360e5909c938 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:45.173765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:45.174344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:45.174652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:45.174935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:45.175698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:45.176011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:45.177498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:45.204472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 604/1243] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:45.214264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:45.356483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:45.402655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4acecd2-90f7-4499-b3b4-3331b5910609 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:45.480965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:45.481933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:45.482328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:45.483950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:45.504414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:45.507900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:29:45.541670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:45.543629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:45.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 640/1244] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:29:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:45.554172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:45.747218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bc0598-1fa0-47d0-ba6f-fc6a7926ed9a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:45.804358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14002e62-463d-44d8-9938-0086843848cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:45.805044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14002e62-463d-44d8-9938-0086843848cf None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:45.805315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14002e62-463d-44d8-9938-0086843848cf None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:45.805591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14002e62-463d-44d8-9938-0086843848cf None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:45.806413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14002e62-463d-44d8-9938-0086843848cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:45.806761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14002e62-463d-44d8-9938-0086843848cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:45.808388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14002e62-463d-44d8-9938-0086843848cf None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:45.833765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 605/1245] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:29:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:45.845002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:46.035366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b927dcab-2a12-4401-86b4-3a6ea29132f3 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:46.089043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b261ae-5f8b-4fa1-bccb-a75849bb23f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:46.089892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b261ae-5f8b-4fa1-bccb-a75849bb23f9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:46.090329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b261ae-5f8b-4fa1-bccb-a75849bb23f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:46.090817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b261ae-5f8b-4fa1-bccb-a75849bb23f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:46.092554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b261ae-5f8b-4fa1-bccb-a75849bb23f9 None admin] RBAC: Adding query filter params (role.id=b032af1c9f7a447a98b75130f5bab37a, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=5aabd82e0fd04345b63c346660e0da54) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:46.093139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b261ae-5f8b-4fa1-bccb-a75849bb23f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:46.096199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b261ae-5f8b-4fa1-bccb-a75849bb23f9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:46.109851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 641/1246] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:29:45 2022] GET /identity/v3/role_assignments?user.id=5aabd82e0fd04345b63c346660e0da54&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=b032af1c9f7a447a98b75130f5bab37a => generated 608 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:49.188058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b42c8-322d-4a2a-93d8-62b555266b26 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:49.189037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b42c8-322d-4a2a-93d8-62b555266b26 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:49.189037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b42c8-322d-4a2a-93d8-62b555266b26 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:49.190295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 606/1247] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:29:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:29:49.197884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc9324-35df-46b3-920e-6a290d5343fe None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:49.198807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc9324-35df-46b3-920e-6a290d5343fe None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:49.199102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc9324-35df-46b3-920e-6a290d5343fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:49.209257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:49.228500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-11bc9324-35df-46b3-920e-6a290d5343fe None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:29:49.230116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-11bc9324-35df-46b3-920e-6a290d5343fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:29:49.346150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 642/1248] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:29:49 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:29:49.373663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27b42c8-322d-4a2a-93d8-62b555266b26 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:49.407848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:49.561913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27b42c8-322d-4a2a-93d8-62b555266b26 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:49.620450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:49.621847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:49.622349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:49.622910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:49.645271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:29:49.661644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:49.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:49.665652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 607/1249] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:29:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:49.673483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bc9324-35df-46b3-920e-6a290d5343fe None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:49.872172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bc9324-35df-46b3-920e-6a290d5343fe None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:49.933898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:49.942304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:49.943220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:49.943572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:49.943924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:49.944787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:49.945255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:49.947239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:49.959416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 643/1250] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:29:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:49.966617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:50.097834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:50.133673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e5bea6-dc51-41c1-abdd-abc6940ec916 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:50.191784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:50.193012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:50.193415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:50.193813 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:50.211913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:50.214346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:50.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:50.246341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:50.246341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:50.246341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:50.246341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:50.246341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 608/1251] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:50.255597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:50.426982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb712b8f-6d7e-42fe-a090-609bac4659fe None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:50.487596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:50.488414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:50.488702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:50.489158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:50.490014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:50.490335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:50.492172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:50.514696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 644/1252] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:50.523124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:50.679974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc57b05f-31ba-4ad5-9991-0483bfad16f8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:50.737217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:50.753182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:50.755003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:50.755556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:50.756254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:50.773635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:50.776894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:29:50.810500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:50.812430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:50.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 609/1253] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:50 2022] GET /identity/v3/users/placement => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:50.823998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:50.936347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:51.019375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dde3b04-6fa5-4bdc-b4f2-b6dc0713b035 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:51.083282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:51.084366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:51.084840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:51.085268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:51.086374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:51.086871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:51.089199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:51.119374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 645/1254] 10.222.0.72 () {56 vars in 1143 bytes} [Mon Jul 11 14:29:50 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:51.129393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:51.296128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d6368f-fd96-4f7c-a02c-76942bbd8fb3 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:51.343842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:51.344547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:51.344841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:51.345250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:51.434200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:51.455774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:51.457700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:51.621642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 610/1255] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:29:51 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 11 14:29:51.629597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:51.777937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51568fc8-74ae-40a2-a562-13c146630bb5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:51.795257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:51.827596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:51.829059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:51.829568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:51.830037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:51.850938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:51.852889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:51.875952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:51.877365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:51.884670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:51.884670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:51.884670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:51.884670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:51.884670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 646/1256] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:51.887380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:52.050061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:52.070514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac21a56-d2d5-4e5e-8da8-2603e2c3bd4d None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:52.133207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:52.134130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:52.134640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:52.135093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:52.136093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:52.136522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:52.138141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:52.158978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 611/1257] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:52.166252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:52.331306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343dec4-4199-446e-8faf-45ee97782bde None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:52.390361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:52.391226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:52.391514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:52.391874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:52.400665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:52.403011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:29:52.418399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:52.419939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:29:52.421349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 647/1258] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:29:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:52.426278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:52.574607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f4bc7a-064f-453b-afe4-4781b18aa76f None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:52.631517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0612eae-1754-41d8-a55e-22bd55665b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:52.632781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0612eae-1754-41d8-a55e-22bd55665b6f None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:52.633457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0612eae-1754-41d8-a55e-22bd55665b6f None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:52.634023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0612eae-1754-41d8-a55e-22bd55665b6f None admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:52.635853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0612eae-1754-41d8-a55e-22bd55665b6f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:52.636315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0612eae-1754-41d8-a55e-22bd55665b6f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:52.639092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0612eae-1754-41d8-a55e-22bd55665b6f None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:52.663854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 612/1259] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:29:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:52.665032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:52.720989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:52.831034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0afd02-25d9-417b-aa4d-7162b306d1d4 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:52.882070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07423d3f-3f6c-4f7d-aa9e-8663dc513353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:52.882405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07423d3f-3f6c-4f7d-aa9e-8663dc513353 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:52.882752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07423d3f-3f6c-4f7d-aa9e-8663dc513353 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:52.882752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07423d3f-3f6c-4f7d-aa9e-8663dc513353 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:52.884375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07423d3f-3f6c-4f7d-aa9e-8663dc513353 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=5aabd82e0fd04345b63c346660e0da54) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:52.884885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07423d3f-3f6c-4f7d-aa9e-8663dc513353 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:52.887151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07423d3f-3f6c-4f7d-aa9e-8663dc513353 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:52.897546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 648/1260] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:29:52 2022] GET /identity/v3/role_assignments?user.id=5aabd82e0fd04345b63c346660e0da54&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 259 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:55.199629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec651c44-294e-4fe5-a008-1dae4879d828 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:55.200212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec651c44-294e-4fe5-a008-1dae4879d828 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:55.200212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec651c44-294e-4fe5-a008-1dae4879d828 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:55.201841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 613/1261] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:29:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:29:55.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4dc56a-b26b-450b-a96a-421b57e85022 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:55.209011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4dc56a-b26b-450b-a96a-421b57e85022 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:55.209189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4dc56a-b26b-450b-a96a-421b57e85022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:55.226255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-0a4dc56a-b26b-450b-a96a-421b57e85022 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:29:55.227536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0a4dc56a-b26b-450b-a96a-421b57e85022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:29:55.302914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 649/1262] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:29:55 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:29:55.329023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec651c44-294e-4fe5-a008-1dae4879d828 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:55.345201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:55.472040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec651c44-294e-4fe5-a008-1dae4879d828 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:55.513417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:55.514288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:55.514645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:55.514932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:55.526355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:29:55.535261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:55.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:55.539120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 614/1263] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:29:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:55.543072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4dc56a-b26b-450b-a96a-421b57e85022 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:55.570358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:55.669970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4dc56a-b26b-450b-a96a-421b57e85022 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:55.717886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151aef61-511e-413e-8655-79b78a0375da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:55.718926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151aef61-511e-413e-8655-79b78a0375da None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:55.719300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151aef61-511e-413e-8655-79b78a0375da None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:55.719701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151aef61-511e-413e-8655-79b78a0375da None admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:55.720804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151aef61-511e-413e-8655-79b78a0375da None admin] RBAC: Adding query filter params (name=admin) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:55.721236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151aef61-511e-413e-8655-79b78a0375da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:55.723285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151aef61-511e-413e-8655-79b78a0375da None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:55.736376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 650/1264] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:29:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:55.742215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:55.816510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:55.853898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e89e75c-9c5d-4654-9935-9f5d2efca634 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:55.892423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:55.893232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:55.893511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:55.893785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:55.905969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:55.907500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:55.927586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:55.929278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:55.931201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:55.931201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:55.931201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:55.931201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:55.931201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 615/1265] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:55.934728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151aef61-511e-413e-8655-79b78a0375da None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:56.025712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:56.047973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151aef61-511e-413e-8655-79b78a0375da None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:56.086339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:56.087108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:56.087438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:56.087764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:56.088548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:56.088899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:56.090337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:56.104404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 651/1266] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:56.111152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:56.234924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe9b4d2-92ed-46fb-82dd-45639b10c241 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:56.255778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:56.272392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:56.273286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:56.273572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:56.273849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] PATH_INFO: `/v3/users/placement` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:56.283242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:56.284624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:29:56.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:56.302362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:56.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 616/1267] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:56 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:56.308863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:56.423896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc875676-e1e6-4163-ab3f-99da7dfa7f3e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:56.462501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:56.463184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:56.463471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:56.463751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:56.464707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:56.465080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:56.466624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:56.476910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:56.487245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 652/1268] 10.222.0.72 () {56 vars in 1143 bytes} [Mon Jul 11 14:29:56 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:56.493859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:56.628321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d10b31-f817-44a2-8198-1b9205c356ea None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:56.667009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:56.667934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:56.668318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:56.668689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:56.680645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:56.682286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:29:56.700302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:56.704977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:29:56.704977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:29:56.704977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:29:56.704977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:56.704977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 617/1269] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:29:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:29:56.708901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:56.823504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90564036-4647-44ff-acb3-1c5ec26c672e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:56.861333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:56.862004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:56.862283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:56.862588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:56.863328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:56.863629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:56.865016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:56.877923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 653/1270] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:29:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:29:56.886070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:56.919705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:57.001747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab6527f-bc79-4bfd-a096-bf76fa80e3db None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:57.052396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:57.053221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:57.053508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:57.053790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:57.062762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:57.064349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:29:57.080333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:29:57.081734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:29:57.084006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 618/1271] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:29:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:29:57.089088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:57.161607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:57.232323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2dfc30-021d-4007-8188-6d70499e87b2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:57.276480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce55850c-3c31-4bed-a350-466204a5b0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:57.277153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce55850c-3c31-4bed-a350-466204a5b0fc None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:57.277480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce55850c-3c31-4bed-a350-466204a5b0fc None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:57.277766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce55850c-3c31-4bed-a350-466204a5b0fc None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:57.278781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce55850c-3c31-4bed-a350-466204a5b0fc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:29:57.279099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce55850c-3c31-4bed-a350-466204a5b0fc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:57.280574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce55850c-3c31-4bed-a350-466204a5b0fc None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:57.294812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 654/1272] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:29:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:29:57.300947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:57.420148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:29:57.431471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37792ab1-2ef2-41b6-b488-b6585e1a6a74 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:57.471516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af679df8-6e1e-4a8a-b9b6-639606145282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:29:57.472430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af679df8-6e1e-4a8a-b9b6-639606145282 None admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:57.472732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af679df8-6e1e-4a8a-b9b6-639606145282 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:57.472992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af679df8-6e1e-4a8a-b9b6-639606145282 None admin] PATH_INFO: `/v3/projects/713b51dac4f240c8b873e83e881c862f/users/5aabd82e0fd04345b63c346660e0da54/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:57.475285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af679df8-6e1e-4a8a-b9b6-639606145282 None admin] RBAC: Authorizing `identity:create_grant(project_id=713b51dac4f240c8b873e83e881c862f, user_id=5aabd82e0fd04345b63c346660e0da54, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:29:57.476857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af679df8-6e1e-4a8a-b9b6-639606145282 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:29:57.493154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 619/1273] 10.222.0.72 () {58 vars in 1439 bytes} [Mon Jul 11 14:29:57 2022] PUT /identity/v3/projects/713b51dac4f240c8b873e83e881c862f/users/5aabd82e0fd04345b63c346660e0da54/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:29:59.689313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac63074b-0873-475a-9e99-3fa89d7104d7 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:59.690115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac63074b-0873-475a-9e99-3fa89d7104d7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:59.690115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac63074b-0873-475a-9e99-3fa89d7104d7 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:59.692254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 655/1274] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:29:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:29:59.700333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c08c26-55a8-463b-a2ae-50ce273005b5 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:29:59.700825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c08c26-55a8-463b-a2ae-50ce273005b5 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:29:59.701270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c08c26-55a8-463b-a2ae-50ce273005b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:29:59.719853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f6c08c26-55a8-463b-a2ae-50ce273005b5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:29:59.721176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f6c08c26-55a8-463b-a2ae-50ce273005b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:29:59.803573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 620/1275] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:29:59 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:29:59.834800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac63074b-0873-475a-9e99-3fa89d7104d7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:29:59.974543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac63074b-0873-475a-9e99-3fa89d7104d7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:29:59.994113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:00.013316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:00.014314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:00.014721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:00.015090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:00.027191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 11 14:30:00.035799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:00.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:00.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 656/1276] 10.222.0.72 () {56 vars in 1096 bytes} [Mon Jul 11 14:29:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:30:00.047045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c08c26-55a8-463b-a2ae-50ce273005b5 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:00.152373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:00.176440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c08c26-55a8-463b-a2ae-50ce273005b5 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:00.220220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:00.221400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:00.221917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:00.222417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:00.223903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:00.224478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:00.227205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:00.242093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 621/1277] 10.222.0.72 () {56 vars in 1099 bytes} [Mon Jul 11 14:30:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:30:00.251543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:00.382956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce83f7a-0043-453e-894b-0a98f6649fa5 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:00.445525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:00.446905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:00.447451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:00.447965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:00.467911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:00.470772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:30:00.504767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:00.505887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:00.506837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:30:00.506837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:30:00.506837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:30:00.506837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:00.507080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 657/1278] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:30:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:30:00.516189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:00.654045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c15bebf-bb14-4810-845c-46ede2288e91 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:00.700801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:00.701854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:00.702193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:00.702571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:00.703433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:00.703895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:00.705344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:00.719508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 622/1279] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:30:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:00.726293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:00.784007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:00.873326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3ea3f-925c-4105-b507-0f5c2b0ddc75 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:00.910404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:00.911347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:00.911671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:00.911983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:00.921357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:00.923526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 11 14:30:00.947448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:00.949517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:00.952044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 658/1280] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:30:00 2022] GET /identity/v3/users/placement => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:30:00.959761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:00.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:01.153159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a53540-2789-4876-83e0-0f24ac062159 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:01.220336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:01.221637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:01.221995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:01.222526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:01.224147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:01.224722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:01.227515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:01.253929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 623/1281] 10.222.0.72 () {56 vars in 1143 bytes} [Mon Jul 11 14:30:00 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:01.262693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:01.418524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:01.437695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050da76a-d67e-42aa-953f-c24d8be594b2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:01.503310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:01.506112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:01.507055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:01.508310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:01.607199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:01.608908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:01.776315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:01.796935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 659/1282] 10.222.0.72 () {58 vars in 1282 bytes} [Mon Jul 11 14:30:01 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 535 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 11 14:30:01.803301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:01.880739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:01.933897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9730becf-fcc1-4b5e-bd90-7b0dd54d777a None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:01.990816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:01.991634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:01.991922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:01.992201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:02.013488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:02.017064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 11 14:30:02.037298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:02.039248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:02.040951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 11 14:30:02.040951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 11 14:30:02.040951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 11 14:30:02.040951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:30:02.040951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 624/1283] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:30:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:30:02.047346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:02.189056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b51255a-6877-4ed7-965e-94e666b372bb None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:02.229033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:02.229722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:02.230009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:02.230287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:02.231068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:02.231382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:02.232788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:02.245974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 660/1284] 10.222.0.72 () {56 vars in 1109 bytes} [Mon Jul 11 14:30:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:02.255350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:02.401157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e07eb-431b-483b-8c2a-4f516836ccd8 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:02.416445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:02.447381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df602631-3371-411b-a408-de4015521c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:02.448090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df602631-3371-411b-a408-de4015521c59 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:02.448371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df602631-3371-411b-a408-de4015521c59 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:02.448646 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df602631-3371-411b-a408-de4015521c59 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:02.456599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df602631-3371-411b-a408-de4015521c59 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:02.458054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df602631-3371-411b-a408-de4015521c59 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application [None req-df602631-3371-411b-a408-de4015521c59 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 11 14:30:02.472308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:02.474277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: ERROR keystone.server.flask.application Jul 11 14:30:02.476399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 625/1285] 10.222.0.72 () {56 vars in 1111 bytes} [Mon Jul 11 14:30:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:30:02.481374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:02.549735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:02.613671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e546033b-ea46-46df-968c-1ed766d07a9e None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:02.655415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce25c3f-a5a6-4dee-ab62-b85214463713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:02.656083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce25c3f-a5a6-4dee-ab62-b85214463713 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:02.656372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce25c3f-a5a6-4dee-ab62-b85214463713 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:02.656651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce25c3f-a5a6-4dee-ab62-b85214463713 None admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:02.657461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce25c3f-a5a6-4dee-ab62-b85214463713 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:02.657773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce25c3f-a5a6-4dee-ab62-b85214463713 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:02.659275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce25c3f-a5a6-4dee-ab62-b85214463713 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:02.673291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 661/1286] 10.222.0.72 () {56 vars in 1148 bytes} [Mon Jul 11 14:30:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:02.679972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df602631-3371-411b-a408-de4015521c59 None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:02.799944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df602631-3371-411b-a408-de4015521c59 None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:02.837345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5714c632-19a2-4f50-9349-96319b6dfc28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:02.838012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714c632-19a2-4f50-9349-96319b6dfc28 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:02.838285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714c632-19a2-4f50-9349-96319b6dfc28 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:02.838601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714c632-19a2-4f50-9349-96319b6dfc28 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:02.839456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5714c632-19a2-4f50-9349-96319b6dfc28 None admin] RBAC: Adding query filter params (role.id=7ce0e1d27bc5490c885bee2563cbd603, scope.project.id=713b51dac4f240c8b873e83e881c862f, user.id=5aabd82e0fd04345b63c346660e0da54) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:02.839774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5714c632-19a2-4f50-9349-96319b6dfc28 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:02.841242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5714c632-19a2-4f50-9349-96319b6dfc28 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:02.848618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 626/1287] 10.222.0.72 () {56 vars in 1374 bytes} [Mon Jul 11 14:30:02 2022] GET /identity/v3/role_assignments?user.id=5aabd82e0fd04345b63c346660e0da54&scope.project.id=713b51dac4f240c8b873e83e881c862f&role.id=7ce0e1d27bc5490c885bee2563cbd603 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:04.979386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec4e91-4887-47ca-ac23-2378c9f999b6 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:04.979908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec4e91-4887-47ca-ac23-2378c9f999b6 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:04.979908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec4e91-4887-47ca-ac23-2378c9f999b6 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:04.981269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 662/1288] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:30:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:30:04.988625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82978902-e7bb-4d64-8e05-03b6481eaa49 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:04.989184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82978902-e7bb-4d64-8e05-03b6481eaa49 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:04.989586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82978902-e7bb-4d64-8e05-03b6481eaa49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:05.011166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-82978902-e7bb-4d64-8e05-03b6481eaa49 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:30:05.012385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-82978902-e7bb-4d64-8e05-03b6481eaa49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:30:05.053735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:05.106070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 627/1289] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:30:04 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 11 14:30:05.130970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fec4e91-4887-47ca-ac23-2378c9f999b6 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:05.254021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fec4e91-4887-47ca-ac23-2378c9f999b6 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:05.267393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:05.293827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:05.294668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:05.294957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:05.295235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:05.295906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:05.297875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:05.311847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:05.313674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:05.314641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 663/1290] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:30:05 2022] GET /identity/v3/services/placement => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:30:05.319446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82978902-e7bb-4d64-8e05-03b6481eaa49 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:05.467386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82978902-e7bb-4d64-8e05-03b6481eaa49 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:05.506958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb66dd3-261b-4045-b9f5-a6920cf3d9bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:05.507746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb66dd3-261b-4045-b9f5-a6920cf3d9bb None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:05.507897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb66dd3-261b-4045-b9f5-a6920cf3d9bb None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:05.508622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb66dd3-261b-4045-b9f5-a6920cf3d9bb None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:05.508817 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb66dd3-261b-4045-b9f5-a6920cf3d9bb None admin] RBAC: Adding query filter params (name=placement) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:05.509135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb66dd3-261b-4045-b9f5-a6920cf3d9bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:05.510562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb66dd3-261b-4045-b9f5-a6920cf3d9bb None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:05.516488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 628/1291] 10.222.0.72 () {56 vars in 1116 bytes} [Mon Jul 11 14:30:05 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:30:05.522539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:05.648811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ba192-2091-46f5-b478-6a7971f537a2 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:05.695869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d4fdd3-2d9a-44a6-b61c-2d9806b52a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:05.696562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d4fdd3-2d9a-44a6-b61c-2d9806b52a52 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:05.696848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d4fdd3-2d9a-44a6-b61c-2d9806b52a52 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:05.697131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d4fdd3-2d9a-44a6-b61c-2d9806b52a52 None admin] PATH_INFO: `/v3/services` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:05.697865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d4fdd3-2d9a-44a6-b61c-2d9806b52a52 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:05.698179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d4fdd3-2d9a-44a6-b61c-2d9806b52a52 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:05.699658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d4fdd3-2d9a-44a6-b61c-2d9806b52a52 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:05.705158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 664/1292] 10.222.0.72 () {56 vars in 1116 bytes} [Mon Jul 11 14:30:05 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:08.027665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6338c3-58f6-4de7-b2b2-bd6f1165acad None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:08.028325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6338c3-58f6-4de7-b2b2-bd6f1165acad None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:08.028613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6338c3-58f6-4de7-b2b2-bd6f1165acad None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:08.031306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 629/1293] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:30:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:30:08.038113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba794ef-7578-42ba-9a82-104fec25ff02 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:08.038738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba794ef-7578-42ba-9a82-104fec25ff02 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:08.039024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba794ef-7578-42ba-9a82-104fec25ff02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:08.057491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-7ba794ef-7578-42ba-9a82-104fec25ff02 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:30:08.058766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7ba794ef-7578-42ba-9a82-104fec25ff02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:30:08.102085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:08.170486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 665/1294] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:30:08 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:30:08.194190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6338c3-58f6-4de7-b2b2-bd6f1165acad None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:08.229920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:08.309474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6338c3-58f6-4de7-b2b2-bd6f1165acad None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:08.358941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263a0b1d-22c0-4e17-90ba-4c9205c2b71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:08.359805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a0b1d-22c0-4e17-90ba-4c9205c2b71e None admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:08.360198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a0b1d-22c0-4e17-90ba-4c9205c2b71e None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:08.360711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a0b1d-22c0-4e17-90ba-4c9205c2b71e None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:08.361507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263a0b1d-22c0-4e17-90ba-4c9205c2b71e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:08.363240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263a0b1d-22c0-4e17-90ba-4c9205c2b71e None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:08.384434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 630/1295] 10.222.0.72 () {60 vars in 1141 bytes} [Mon Jul 11 14:30:08 2022] POST /identity/v3/services => generated 246 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:30:10.512772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7d89b-157a-46dc-8510-db10573aee0b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:10.513353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7d89b-157a-46dc-8510-db10573aee0b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:10.513353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7d89b-157a-46dc-8510-db10573aee0b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:10.515193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 666/1296] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:30:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:30:10.521837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212be27-27b5-49f6-b0e0-c6491247278f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:10.522333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212be27-27b5-49f6-b0e0-c6491247278f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:10.522447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212be27-27b5-49f6-b0e0-c6491247278f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:10.540215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-5212be27-27b5-49f6-b0e0-c6491247278f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:30:10.541430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-5212be27-27b5-49f6-b0e0-c6491247278f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:30:10.622737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 631/1297] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:30:10 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:30:10.645640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba7d89b-157a-46dc-8510-db10573aee0b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:10.756833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba7d89b-157a-46dc-8510-db10573aee0b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:10.794675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:10.795401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:10.795688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:10.795967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:10.796664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:10.798055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:10.810972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:10.812367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:10.813293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 667/1298] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:30:10 2022] GET /identity/v3/services/placement => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 11 14:30:10.819061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212be27-27b5-49f6-b0e0-c6491247278f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:10.841998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:10.926897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212be27-27b5-49f6-b0e0-c6491247278f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:10.962886 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:10.963560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:10.963849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:10.964126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:10.964849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] RBAC: Adding query filter params (name=placement) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:10.965161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:10.966580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:10.972681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 632/1299] 10.222.0.72 () {56 vars in 1116 bytes} [Mon Jul 11 14:30:10 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:10.978737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:11.012004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:11.089061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccbcfb4-999b-4a89-a494-714169fa8fb0 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:11.123421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d95c24-a417-483d-9641-c0ba53f998e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:11.124088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d95c24-a417-483d-9641-c0ba53f998e9 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:11.124875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d95c24-a417-483d-9641-c0ba53f998e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:11.125158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d95c24-a417-483d-9641-c0ba53f998e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:11.125965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d95c24-a417-483d-9641-c0ba53f998e9 None admin] RBAC: Adding query filter params (interface=public, service_id=58f6022159ce44968e8fb06daa6074fc, region_id=RegionOne) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:11.126281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d95c24-a417-483d-9641-c0ba53f998e9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:11.127703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d95c24-a417-483d-9641-c0ba53f998e9 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:11.133151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 668/1300] 10.222.0.72 () {56 vars in 1251 bytes} [Mon Jul 11 14:30:10 2022] GET /identity/v3/endpoints?service_id=58f6022159ce44968e8fb06daa6074fc&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:11.140658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:11.229574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:11.253880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5e137-de34-4ade-bcaf-42963ecc35eb None admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:11.316389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb82c49b-dfac-485d-aa4e-ba408e4f4e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:11.317258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82c49b-dfac-485d-aa4e-ba408e4f4e29 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:11.317705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82c49b-dfac-485d-aa4e-ba408e4f4e29 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:11.318073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82c49b-dfac-485d-aa4e-ba408e4f4e29 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:11.318871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb82c49b-dfac-485d-aa4e-ba408e4f4e29 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:11.320828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb82c49b-dfac-485d-aa4e-ba408e4f4e29 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:11.332016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 633/1301] 10.222.0.72 () {56 vars in 1087 bytes} [Mon Jul 11 14:30:11 2022] GET /identity/v3/services => generated 2191 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:30:13.439799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c29d22b-87ee-4ca3-ab13-e7b94fb94b73 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:13.440372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c29d22b-87ee-4ca3-ab13-e7b94fb94b73 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:13.440372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c29d22b-87ee-4ca3-ab13-e7b94fb94b73 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:13.441745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 669/1302] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:30:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:30:13.448850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44f088c-92db-4c70-9597-654a2e932113 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:13.449437 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44f088c-92db-4c70-9597-654a2e932113 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:13.449859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44f088c-92db-4c70-9597-654a2e932113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:13.479464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b44f088c-92db-4c70-9597-654a2e932113 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:30:13.481531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b44f088c-92db-4c70-9597-654a2e932113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:30:13.603732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 634/1303] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:30:13 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 11 14:30:13.627851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c29d22b-87ee-4ca3-ab13-e7b94fb94b73 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:13.744315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:13.751924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c29d22b-87ee-4ca3-ab13-e7b94fb94b73 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:13.798113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:13.798986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:13.799271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:13.799546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:13.800210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:13.801633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 11 14:30:13.814762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 11 14:30:13.816946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.get_or_create( Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application with Lock( Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self._enter() Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application return self.creator() Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application created_value = creator( Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: ERROR keystone.server.flask.application Jul 11 14:30:13.819211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 670/1304] 10.222.0.72 () {56 vars in 1117 bytes} [Mon Jul 11 14:30:13 2022] GET /identity/v3/services/placement => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 11 14:30:13.825196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44f088c-92db-4c70-9597-654a2e932113 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:13.955503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:13.981937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44f088c-92db-4c70-9597-654a2e932113 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:14.022247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2037e3d3-b9d4-419b-bff3-76d7aea3da18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:14.022875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037e3d3-b9d4-419b-bff3-76d7aea3da18 None admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:14.023162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037e3d3-b9d4-419b-bff3-76d7aea3da18 None admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:14.023445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037e3d3-b9d4-419b-bff3-76d7aea3da18 None admin] PATH_INFO: `/v3/services` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:14.024183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2037e3d3-b9d4-419b-bff3-76d7aea3da18 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:30:14.024505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2037e3d3-b9d4-419b-bff3-76d7aea3da18 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:14.025927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2037e3d3-b9d4-419b-bff3-76d7aea3da18 None admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:14.032458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 635/1305] 10.222.0.72 () {56 vars in 1116 bytes} [Mon Jul 11 14:30:13 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:30:14.038530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:30:14.163003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8ebb77-dc57-4e14-8f62-c6c3f392ae35 None admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:30:14.213081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4b4b25-a56d-47eb-9394-a2daa3fb1984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:30:14.214223 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b4b25-a56d-47eb-9394-a2daa3fb1984 None admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:30:14.214772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b4b25-a56d-47eb-9394-a2daa3fb1984 None admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:30:14.215279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b4b25-a56d-47eb-9394-a2daa3fb1984 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:30:14.216684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4b4b25-a56d-47eb-9394-a2daa3fb1984 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:30:14.219742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4b4b25-a56d-47eb-9394-a2daa3fb1984 None admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:30:14.246294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 671/1306] 10.222.0.72 () {60 vars in 1144 bytes} [Mon Jul 11 14:30:14 2022] POST /identity/v3/endpoints => generated 331 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:30:24.029154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=30 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:30:24.241442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:26.075699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1784d5b1-c82d-49d5-888f-c06a199e843a None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:26.076818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1784d5b1-c82d-49d5-888f-c06a199e843a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:26.076818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1784d5b1-c82d-49d5-888f-c06a199e843a None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:26.084708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9fa015-8866-450f-b8c8-4658ed3a8cb2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:26.085300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9fa015-8866-450f-b8c8-4658ed3a8cb2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:26.085541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9fa015-8866-450f-b8c8-4658ed3a8cb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:26.090612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 636/1307] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:26 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:26.245869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2c9fa015-8866-450f-b8c8-4658ed3a8cb2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:26.249112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2c9fa015-8866-450f-b8c8-4658ed3a8cb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:26.576749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 672/1308] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 493 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:31:26.668102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149298b-fbb3-4692-a565-2f3d0a1c24d1 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:26.668753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149298b-fbb3-4692-a565-2f3d0a1c24d1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:26.669957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149298b-fbb3-4692-a565-2f3d0a1c24d1 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:26.674105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 637/1309] 10.222.0.72 () {54 vars in 964 bytes} [Mon Jul 11 14:31:26 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:26.686277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0ed84e-e47b-4010-886c-9c3e84b26fbd None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:26.688760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0ed84e-e47b-4010-886c-9c3e84b26fbd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:26.690339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0ed84e-e47b-4010-886c-9c3e84b26fbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:26.939706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1d0ed84e-e47b-4010-886c-9c3e84b26fbd None None] MFA Rules not processed for user `ac5254fb5eff4108be9ac3992e05a99c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:26.941274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1d0ed84e-e47b-4010-886c-9c3e84b26fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:26.996077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:27.101879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 673/1310] 10.222.0.72 () {58 vars in 1061 bytes} [Mon Jul 11 14:31:26 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 421 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:31:27.110326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149298b-fbb3-4692-a565-2f3d0a1c24d1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:27.112937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f149298b-fbb3-4692-a565-2f3d0a1c24d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:27.288084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149298b-fbb3-4692-a565-2f3d0a1c24d1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:27.397749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139021a6-f6fb-4409-8856-0887a29308d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:27.398710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139021a6-f6fb-4409-8856-0887a29308d4 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:27.399157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139021a6-f6fb-4409-8856-0887a29308d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:27.399660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139021a6-f6fb-4409-8856-0887a29308d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:27.402827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-139021a6-f6fb-4409-8856-0887a29308d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:27.475588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:27.539959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139021a6-f6fb-4409-8856-0887a29308d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:27.541699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139021a6-f6fb-4409-8856-0887a29308d4 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:27.586956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 638/1311] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:30.129962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbbdaef-838a-4b2a-b7ec-8bee90ccac8a None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:30.130535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbbdaef-838a-4b2a-b7ec-8bee90ccac8a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:30.130893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbbdaef-838a-4b2a-b7ec-8bee90ccac8a None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:30.132455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 674/1312] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:30.139529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f688c6d0-246d-4d3b-bf83-5ddc57b4d1ee None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:30.140060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f688c6d0-246d-4d3b-bf83-5ddc57b4d1ee None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:30.140060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f688c6d0-246d-4d3b-bf83-5ddc57b4d1ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:30.223006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f688c6d0-246d-4d3b-bf83-5ddc57b4d1ee None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:30.224519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f688c6d0-246d-4d3b-bf83-5ddc57b4d1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:30.340483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 639/1313] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:31:30.393415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160f5ce2-cc71-4aca-8bcf-3c6395e56a91 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:30.393925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160f5ce2-cc71-4aca-8bcf-3c6395e56a91 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:30.394137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160f5ce2-cc71-4aca-8bcf-3c6395e56a91 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:30.395816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 675/1314] 10.222.0.72 () {54 vars in 964 bytes} [Mon Jul 11 14:31:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:30.402713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be53077-ee58-402c-af78-4a74480a866d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:30.403012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be53077-ee58-402c-af78-4a74480a866d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:30.403900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be53077-ee58-402c-af78-4a74480a866d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:30.415220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:30.494914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8be53077-ee58-402c-af78-4a74480a866d None None] MFA Rules not processed for user `ac5254fb5eff4108be9ac3992e05a99c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:30.496148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8be53077-ee58-402c-af78-4a74480a866d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:30.577243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 640/1315] 10.222.0.72 () {58 vars in 1061 bytes} [Mon Jul 11 14:31:30 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:30.587054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160f5ce2-cc71-4aca-8bcf-3c6395e56a91 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:30.588653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-160f5ce2-cc71-4aca-8bcf-3c6395e56a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:30.697940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160f5ce2-cc71-4aca-8bcf-3c6395e56a91 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:30.773707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:30.776870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a699978b-875b-4221-8026-d455662aab33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:30.777944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a699978b-875b-4221-8026-d455662aab33 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:30.778332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a699978b-875b-4221-8026-d455662aab33 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:30.778680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a699978b-875b-4221-8026-d455662aab33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:30.781048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a699978b-875b-4221-8026-d455662aab33 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:30.895606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a699978b-875b-4221-8026-d455662aab33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:30.897046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a699978b-875b-4221-8026-d455662aab33 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:30.915613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 676/1316] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:33.489337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aff985-726d-4c81-94f9-7e88a3048971 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:33.489921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aff985-726d-4c81-94f9-7e88a3048971 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:33.489921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aff985-726d-4c81-94f9-7e88a3048971 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:33.491496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 641/1317] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:31:33.497776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1c63b-7345-42c8-8b94-127e37c2f722 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:33.498250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1c63b-7345-42c8-8b94-127e37c2f722 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:33.498358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1c63b-7345-42c8-8b94-127e37c2f722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:33.570397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e6c1c63b-7345-42c8-8b94-127e37c2f722 None None] MFA Rules not processed for user `a6d8547cc7c84d159b1e737f172c73fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:33.571699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e6c1c63b-7345-42c8-8b94-127e37c2f722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:33.628537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:33.678819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 677/1318] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:33 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:33.718966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aff985-726d-4c81-94f9-7e88a3048971 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:33.720513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f7aff985-726d-4c81-94f9-7e88a3048971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:33.829232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aff985-726d-4c81-94f9-7e88a3048971 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:33.832057 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:33.943943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d3775e-6d4a-4989-824f-43a56baeed6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:33.944605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d3775e-6d4a-4989-824f-43a56baeed6b service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:33.944896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d3775e-6d4a-4989-824f-43a56baeed6b service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:33.945177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d3775e-6d4a-4989-824f-43a56baeed6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:33.946910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d5d3775e-6d4a-4989-824f-43a56baeed6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:34.061286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d3775e-6d4a-4989-824f-43a56baeed6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:34.062970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d3775e-6d4a-4989-824f-43a56baeed6b service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:34.082646 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 642/1319] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:33 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:36.880141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a0992a-1a21-4f38-adb6-062c9f990b20 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:36.880837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a0992a-1a21-4f38-adb6-062c9f990b20 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:36.880837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a0992a-1a21-4f38-adb6-062c9f990b20 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:36.882120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 678/1320] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:36.887751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba182e1-1e89-497e-a0c3-fc1172086f32 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:36.888277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba182e1-1e89-497e-a0c3-fc1172086f32 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:36.888372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba182e1-1e89-497e-a0c3-fc1172086f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:36.933133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:36.963468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bba182e1-1e89-497e-a0c3-fc1172086f32 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:36.964733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bba182e1-1e89-497e-a0c3-fc1172086f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:37.064975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 643/1321] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:36 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:37.106112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a0992a-1a21-4f38-adb6-062c9f990b20 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:37.107817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-22a0992a-1a21-4f38-adb6-062c9f990b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:37.223948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a0992a-1a21-4f38-adb6-062c9f990b20 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:37.302122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d965fa59-396b-4876-8d9d-30ad111dce38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:37.302911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d965fa59-396b-4876-8d9d-30ad111dce38 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:37.303094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d965fa59-396b-4876-8d9d-30ad111dce38 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:37.303374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d965fa59-396b-4876-8d9d-30ad111dce38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:37.305052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d965fa59-396b-4876-8d9d-30ad111dce38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:37.313117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:37.422241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d965fa59-396b-4876-8d9d-30ad111dce38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:37.423737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d965fa59-396b-4876-8d9d-30ad111dce38 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:37.443344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 679/1322] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:37 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:40.246209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f0b0ce-57bd-480b-a838-de1b9fbcb5d1 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:40.246898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f0b0ce-57bd-480b-a838-de1b9fbcb5d1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:40.246898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f0b0ce-57bd-480b-a838-de1b9fbcb5d1 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:40.248224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 644/1323] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:40.253591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d17f4ec-f85a-4f6c-91b0-d453a86ec941 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:40.254023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d17f4ec-f85a-4f6c-91b0-d453a86ec941 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:40.254287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d17f4ec-f85a-4f6c-91b0-d453a86ec941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:40.322687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4d17f4ec-f85a-4f6c-91b0-d453a86ec941 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:40.323910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4d17f4ec-f85a-4f6c-91b0-d453a86ec941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:40.387106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:40.434812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 680/1324] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:40.471806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f0b0ce-57bd-480b-a838-de1b9fbcb5d1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:40.473386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-20f0b0ce-57bd-480b-a838-de1b9fbcb5d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:40.585985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f0b0ce-57bd-480b-a838-de1b9fbcb5d1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:40.600394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:40.665971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e944647-5a55-4f0d-9032-4eeec3ef604e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:40.666670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e944647-5a55-4f0d-9032-4eeec3ef604e service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:40.666956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e944647-5a55-4f0d-9032-4eeec3ef604e service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:40.667237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e944647-5a55-4f0d-9032-4eeec3ef604e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:40.669529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4e944647-5a55-4f0d-9032-4eeec3ef604e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:40.781355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e944647-5a55-4f0d-9032-4eeec3ef604e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:40.782857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e944647-5a55-4f0d-9032-4eeec3ef604e service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:40.801704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 645/1325] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:42.903578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee5b0f-cb98-46f4-9d1f-501bb40ce5c0 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:42.904132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee5b0f-cb98-46f4-9d1f-501bb40ce5c0 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:42.904132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee5b0f-cb98-46f4-9d1f-501bb40ce5c0 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:42.905902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 681/1326] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:42.911208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9876a-73ed-41a8-ac92-9dc0a19d4bd3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:42.911679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9876a-73ed-41a8-ac92-9dc0a19d4bd3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:42.911799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9876a-73ed-41a8-ac92-9dc0a19d4bd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:42.960315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:42.979343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-76b9876a-73ed-41a8-ac92-9dc0a19d4bd3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:42.980573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-76b9876a-73ed-41a8-ac92-9dc0a19d4bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:43.082325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 646/1327] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:43.118391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ee5b0f-cb98-46f4-9d1f-501bb40ce5c0 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:43.119986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a0ee5b0f-cb98-46f4-9d1f-501bb40ce5c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:43.232377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ee5b0f-cb98-46f4-9d1f-501bb40ce5c0 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:43.308640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9124e1a-899c-4c9e-816c-dde0c08b554f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:43.309864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9124e1a-899c-4c9e-816c-dde0c08b554f service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:43.310148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9124e1a-899c-4c9e-816c-dde0c08b554f service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:43.310443 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9124e1a-899c-4c9e-816c-dde0c08b554f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:43.312266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e9124e1a-899c-4c9e-816c-dde0c08b554f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:43.332828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:43.435737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9124e1a-899c-4c9e-816c-dde0c08b554f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:43.437247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9124e1a-899c-4c9e-816c-dde0c08b554f service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:43.457094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 682/1328] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:45.572837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1eeb4a-1581-4c24-a0dc-509494740354 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:45.573397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1eeb4a-1581-4c24-a0dc-509494740354 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:45.573397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1eeb4a-1581-4c24-a0dc-509494740354 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:45.575492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 647/1329] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:45.580704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ade85bc-8dee-4059-a9aa-277f7b4105c9 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:45.581138 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ade85bc-8dee-4059-a9aa-277f7b4105c9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:45.581388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ade85bc-8dee-4059-a9aa-277f7b4105c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:45.648916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-8ade85bc-8dee-4059-a9aa-277f7b4105c9 None None] MFA Rules not processed for user `a6d8547cc7c84d159b1e737f172c73fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:45.650078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8ade85bc-8dee-4059-a9aa-277f7b4105c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:45.721759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:45.756366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 683/1330] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:45 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:45.804325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1eeb4a-1581-4c24-a0dc-509494740354 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:45.805912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1a1eeb4a-1581-4c24-a0dc-509494740354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:45.910754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1eeb4a-1581-4c24-a0dc-509494740354 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:45.931435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:45.988637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75270def-c21c-484d-aeae-5f725623dbc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:45.989377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75270def-c21c-484d-aeae-5f725623dbc0 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:45.989666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75270def-c21c-484d-aeae-5f725623dbc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:45.989946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75270def-c21c-484d-aeae-5f725623dbc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:45.991701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-75270def-c21c-484d-aeae-5f725623dbc0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:46.113108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75270def-c21c-484d-aeae-5f725623dbc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:46.114607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75270def-c21c-484d-aeae-5f725623dbc0 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:46.135447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 648/1331] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:45 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:48.603210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b53f1ab-2205-40df-b5ca-3ef8bb228481 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:48.603734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b53f1ab-2205-40df-b5ca-3ef8bb228481 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:48.603849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b53f1ab-2205-40df-b5ca-3ef8bb228481 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:48.605153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 684/1332] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:48.611174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80399be-41d2-434f-b794-35bbb52f48dd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:48.612041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80399be-41d2-434f-b794-35bbb52f48dd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:48.612041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80399be-41d2-434f-b794-35bbb52f48dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:48.679648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:48.682767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d80399be-41d2-434f-b794-35bbb52f48dd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:48.685025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d80399be-41d2-434f-b794-35bbb52f48dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:48.786110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 649/1333] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:48.822794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b53f1ab-2205-40df-b5ca-3ef8bb228481 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:48.824415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6b53f1ab-2205-40df-b5ca-3ef8bb228481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:48.933202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b53f1ab-2205-40df-b5ca-3ef8bb228481 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:49.015152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a906e19c-d05d-49b8-8f24-c4e516536c8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:49.015814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a906e19c-d05d-49b8-8f24-c4e516536c8d service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:49.016100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a906e19c-d05d-49b8-8f24-c4e516536c8d service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:49.016377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a906e19c-d05d-49b8-8f24-c4e516536c8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:49.018194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a906e19c-d05d-49b8-8f24-c4e516536c8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:49.047846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:49.140558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a906e19c-d05d-49b8-8f24-c4e516536c8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:49.142019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a906e19c-d05d-49b8-8f24-c4e516536c8d service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:49.160018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 685/1334] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:51.320972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1d271-9d97-4bc8-80ad-bb7564b2e2d9 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:51.321553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1d271-9d97-4bc8-80ad-bb7564b2e2d9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:51.321553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1d271-9d97-4bc8-80ad-bb7564b2e2d9 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:51.323048 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 650/1335] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:51.328747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5258c-2434-48da-b1c5-fabdefc59fd4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:51.329212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5258c-2434-48da-b1c5-fabdefc59fd4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:51.329305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5258c-2434-48da-b1c5-fabdefc59fd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:51.399542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c8a5258c-2434-48da-b1c5-fabdefc59fd4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:51.400775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c8a5258c-2434-48da-b1c5-fabdefc59fd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:51.484287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:51.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 686/1336] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:51.560134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e1d271-9d97-4bc8-80ad-bb7564b2e2d9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:51.561723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-54e1d271-9d97-4bc8-80ad-bb7564b2e2d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:51.681423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e1d271-9d97-4bc8-80ad-bb7564b2e2d9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:51.711747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:51.760234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1e5268-1a61-4b8d-90f5-2e48b0506bbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:51.760891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e5268-1a61-4b8d-90f5-2e48b0506bbf service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:51.761177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e5268-1a61-4b8d-90f5-2e48b0506bbf service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:51.761461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e5268-1a61-4b8d-90f5-2e48b0506bbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:51.763237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3f1e5268-1a61-4b8d-90f5-2e48b0506bbf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:51.877241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1e5268-1a61-4b8d-90f5-2e48b0506bbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:51.878799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1e5268-1a61-4b8d-90f5-2e48b0506bbf service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:51.897195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 651/1337] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:31:54.429853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e44dcd-5778-4146-9872-528199269ef5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:54.430509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e44dcd-5778-4146-9872-528199269ef5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:54.430509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e44dcd-5778-4146-9872-528199269ef5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:54.431816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 687/1338] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:54.437854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf8705-e886-498e-9130-b14a1a881205 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:54.438477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf8705-e886-498e-9130-b14a1a881205 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:54.438477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf8705-e886-498e-9130-b14a1a881205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:54.512227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-24cf8705-e886-498e-9130-b14a1a881205 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:54.513484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-24cf8705-e886-498e-9130-b14a1a881205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:54.523282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:54.620023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 652/1339] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:54 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:54.655962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e44dcd-5778-4146-9872-528199269ef5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:54.657561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-76e44dcd-5778-4146-9872-528199269ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:54.762283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e44dcd-5778-4146-9872-528199269ef5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:54.836536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f18f939-f203-41f1-92db-2f5213b9d71b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:54.837190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f18f939-f203-41f1-92db-2f5213b9d71b service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:54.837474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f18f939-f203-41f1-92db-2f5213b9d71b service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:54.837748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f18f939-f203-41f1-92db-2f5213b9d71b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:54.839475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1f18f939-f203-41f1-92db-2f5213b9d71b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:54.877757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:54.955249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f18f939-f203-41f1-92db-2f5213b9d71b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:54.956612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f18f939-f203-41f1-92db-2f5213b9d71b service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:54.975004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 688/1340] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:54 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:31:59.124360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dade0a-54de-458a-b298-15b30233ad8b None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:59.124850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dade0a-54de-458a-b298-15b30233ad8b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:59.124963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dade0a-54de-458a-b298-15b30233ad8b None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:59.126304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 653/1341] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:31:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:31:59.131920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcce6721-91dc-4b22-a697-1b14afbae64a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:59.132533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcce6721-91dc-4b22-a697-1b14afbae64a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:59.132533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcce6721-91dc-4b22-a697-1b14afbae64a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:59.203211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-bcce6721-91dc-4b22-a697-1b14afbae64a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:31:59.204451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bcce6721-91dc-4b22-a697-1b14afbae64a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:59.298582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:59.310686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 689/1342] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:31:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:31:59.349815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dade0a-54de-458a-b298-15b30233ad8b None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:31:59.351421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-09dade0a-54de-458a-b298-15b30233ad8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:59.470809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dade0a-54de-458a-b298-15b30233ad8b None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:31:59.517349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:31:59.558921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354f1544-0d77-4be3-9d8d-de2c3eee5551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:31:59.559501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354f1544-0d77-4be3-9d8d-de2c3eee5551 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:31:59.559780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354f1544-0d77-4be3-9d8d-de2c3eee5551 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:31:59.560059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354f1544-0d77-4be3-9d8d-de2c3eee5551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:31:59.561812 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-354f1544-0d77-4be3-9d8d-de2c3eee5551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:31:59.705263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354f1544-0d77-4be3-9d8d-de2c3eee5551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:31:59.706732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354f1544-0d77-4be3-9d8d-de2c3eee5551 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:31:59.726155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 654/1343] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:31:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:02.571375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b279bd93-da27-41ba-b0f3-64e8cd65b17d None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:02.571951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b279bd93-da27-41ba-b0f3-64e8cd65b17d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:02.571951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b279bd93-da27-41ba-b0f3-64e8cd65b17d None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:02.573462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 690/1344] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:02.580010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4693d-496f-49a0-81fe-e5ed11f903b8 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:02.580480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4693d-496f-49a0-81fe-e5ed11f903b8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:02.580594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4693d-496f-49a0-81fe-e5ed11f903b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:02.648445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-dbe4693d-496f-49a0-81fe-e5ed11f903b8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:02.649579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dbe4693d-496f-49a0-81fe-e5ed11f903b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:02.673450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:02.759008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 655/1345] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:02 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:02.797334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b279bd93-da27-41ba-b0f3-64e8cd65b17d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:02.799008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b279bd93-da27-41ba-b0f3-64e8cd65b17d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:02.906992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b279bd93-da27-41ba-b0f3-64e8cd65b17d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:02.981289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5645fe-eae8-4a1e-af69-6f285131c725 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:02.981963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5645fe-eae8-4a1e-af69-6f285131c725 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:02.982234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5645fe-eae8-4a1e-af69-6f285131c725 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:02.982545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5645fe-eae8-4a1e-af69-6f285131c725 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:02.984191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ef5645fe-eae8-4a1e-af69-6f285131c725 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:03.028880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:03.104386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5645fe-eae8-4a1e-af69-6f285131c725 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:03.105904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5645fe-eae8-4a1e-af69-6f285131c725 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:03.127648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 691/1346] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:02 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:07.340603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f332b1f-ffe5-40a2-b586-e157ed008430 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:07.341167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f332b1f-ffe5-40a2-b586-e157ed008430 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:07.341259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f332b1f-ffe5-40a2-b586-e157ed008430 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:07.342945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 656/1347] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:07.348696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2a67d-48ca-4427-b0be-1bde65f5c569 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:07.349229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2a67d-48ca-4427-b0be-1bde65f5c569 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:07.349327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2a67d-48ca-4427-b0be-1bde65f5c569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:07.423123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-3bc2a67d-48ca-4427-b0be-1bde65f5c569 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:07.424390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3bc2a67d-48ca-4427-b0be-1bde65f5c569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:07.526652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:07.529337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 692/1348] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:07.575374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f332b1f-ffe5-40a2-b586-e157ed008430 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:07.577333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3f332b1f-ffe5-40a2-b586-e157ed008430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:07.695699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f332b1f-ffe5-40a2-b586-e157ed008430 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:07.748644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:07.776894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8335315e-046f-45b0-bf37-650310f805a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:07.777590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8335315e-046f-45b0-bf37-650310f805a6 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:07.777874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8335315e-046f-45b0-bf37-650310f805a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:07.778149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8335315e-046f-45b0-bf37-650310f805a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:07.779863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8335315e-046f-45b0-bf37-650310f805a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:07.897487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8335315e-046f-45b0-bf37-650310f805a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:07.899015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8335315e-046f-45b0-bf37-650310f805a6 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:07.919088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 657/1349] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:13.295837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a637c2bd-5551-4f7a-9fa6-bdb765d9422c None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:13.295837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a637c2bd-5551-4f7a-9fa6-bdb765d9422c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:13.295837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a637c2bd-5551-4f7a-9fa6-bdb765d9422c None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:13.296909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 693/1350] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:32:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:13.302575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14807f4-4917-497a-96a1-023b32ac16a3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:13.303221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14807f4-4917-497a-96a1-023b32ac16a3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:13.303221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14807f4-4917-497a-96a1-023b32ac16a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:13.453837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d14807f4-4917-497a-96a1-023b32ac16a3 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:13.454965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d14807f4-4917-497a-96a1-023b32ac16a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:13.478833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:13.631136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 658/1351] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:32:13 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:19.044638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b2c41-bc04-4e29-9132-91b5217c4ef5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:19.046165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b2c41-bc04-4e29-9132-91b5217c4ef5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:19.046692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b2c41-bc04-4e29-9132-91b5217c4ef5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:19.049602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 694/1352] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:32:19 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:19.058488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68bde1b-f879-4cb1-bb9f-2cb437bb5170 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:19.059579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68bde1b-f879-4cb1-bb9f-2cb437bb5170 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:19.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68bde1b-f879-4cb1-bb9f-2cb437bb5170 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:19.201537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c68bde1b-f879-4cb1-bb9f-2cb437bb5170 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:19.203527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c68bde1b-f879-4cb1-bb9f-2cb437bb5170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:19.388403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:19.472574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 659/1353] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:32:19 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 415 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:32:20.596759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad263df-b9e2-4b68-a19a-26335255fb1d None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:20.597279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad263df-b9e2-4b68-a19a-26335255fb1d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:20.597688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad263df-b9e2-4b68-a19a-26335255fb1d None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:20.600274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 695/1354] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:32:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:20.605829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8456ecc2-c1dd-4c3d-aa67-36f24d8a1895 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:20.606390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8456ecc2-c1dd-4c3d-aa67-36f24d8a1895 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:20.606390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8456ecc2-c1dd-4c3d-aa67-36f24d8a1895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:20.713163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8456ecc2-c1dd-4c3d-aa67-36f24d8a1895 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:20.715234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8456ecc2-c1dd-4c3d-aa67-36f24d8a1895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:20.865853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 660/1355] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:32:20 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:24.716474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0996fb6-9e57-446d-8526-4baf32b50b62 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:24.717186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0996fb6-9e57-446d-8526-4baf32b50b62 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:24.717616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0996fb6-9e57-446d-8526-4baf32b50b62 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:24.720326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 696/1356] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:24.731271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02496520-a6c6-4e81-817a-fe448f14a149 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:24.731869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02496520-a6c6-4e81-817a-fe448f14a149 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:24.732380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02496520-a6c6-4e81-817a-fe448f14a149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:24.785487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:24.827735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-02496520-a6c6-4e81-817a-fe448f14a149 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:24.829350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-02496520-a6c6-4e81-817a-fe448f14a149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:25.066390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 661/1357] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:25.163620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5258633-637c-4470-85c2-1d6dc30c1dde None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:25.165088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5258633-637c-4470-85c2-1d6dc30c1dde None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:25.165578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5258633-637c-4470-85c2-1d6dc30c1dde None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:25.172666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 697/1358] 10.222.0.72 () {54 vars in 961 bytes} [Mon Jul 11 14:32:25 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:25.192155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eb98a1-bc59-498a-ae9f-af324f3daff0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:25.193449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eb98a1-bc59-498a-ae9f-af324f3daff0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:25.193763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eb98a1-bc59-498a-ae9f-af324f3daff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:25.352757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e7eb98a1-bc59-498a-ae9f-af324f3daff0 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:25.355220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e7eb98a1-bc59-498a-ae9f-af324f3daff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:25.428672 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:25.578638 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 662/1359] 10.222.0.72 () {58 vars in 1058 bytes} [Mon Jul 11 14:32:25 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 388 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:25.590997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5258633-637c-4470-85c2-1d6dc30c1dde None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:25.594503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c5258633-637c-4470-85c2-1d6dc30c1dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:25.828835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5258633-637c-4470-85c2-1d6dc30c1dde None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:25.954319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:26.008961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9247c6d7-1ba6-4d01-8a3b-922b19c3a16b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:26.009940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247c6d7-1ba6-4d01-8a3b-922b19c3a16b service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:26.010323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247c6d7-1ba6-4d01-8a3b-922b19c3a16b service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:26.011325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247c6d7-1ba6-4d01-8a3b-922b19c3a16b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:26.013642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9247c6d7-1ba6-4d01-8a3b-922b19c3a16b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:26.166587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9247c6d7-1ba6-4d01-8a3b-922b19c3a16b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:26.167201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9247c6d7-1ba6-4d01-8a3b-922b19c3a16b service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:26.190711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 698/1360] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 600 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:29.125109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1828f499-b314-423c-ae9a-ab1eddf6b03f None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:29.125609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1828f499-b314-423c-ae9a-ab1eddf6b03f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:29.126473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1828f499-b314-423c-ae9a-ab1eddf6b03f None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:29.147112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 663/1361] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:29 2022] GET /identity => generated 270 bytes in 20 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:29.147344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cf34c7-4a19-4312-8fd6-ff6b1d439195 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:29.147344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cf34c7-4a19-4312-8fd6-ff6b1d439195 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:29.147344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cf34c7-4a19-4312-8fd6-ff6b1d439195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:29.262512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-39cf34c7-4a19-4312-8fd6-ff6b1d439195 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:29.264229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-39cf34c7-4a19-4312-8fd6-ff6b1d439195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:29.315801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:29.437609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 699/1362] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:29.508653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1828f499-b314-423c-ae9a-ab1eddf6b03f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:29.511097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1828f499-b314-423c-ae9a-ab1eddf6b03f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:29.815321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1828f499-b314-423c-ae9a-ab1eddf6b03f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:29.827520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:29.982804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2c9439-617b-4a56-baa3-eb029a8dbd10 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:29.983482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2c9439-617b-4a56-baa3-eb029a8dbd10 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:29.983839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2c9439-617b-4a56-baa3-eb029a8dbd10 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:29.984177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2c9439-617b-4a56-baa3-eb029a8dbd10 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:29.989673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4b2c9439-617b-4a56-baa3-eb029a8dbd10 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:30.131837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2c9439-617b-4a56-baa3-eb029a8dbd10 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:30.134076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2c9439-617b-4a56-baa3-eb029a8dbd10 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:30.163703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 664/1363] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 656 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:32.752536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403cc4b3-18dd-40de-9bd8-f71da12b97b4 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:32.753072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403cc4b3-18dd-40de-9bd8-f71da12b97b4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:32.753072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403cc4b3-18dd-40de-9bd8-f71da12b97b4 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:32.756284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 700/1364] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:32.763855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7409a6c3-0220-4cf4-9a0e-df2114eb8de4 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:32.764708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7409a6c3-0220-4cf4-9a0e-df2114eb8de4 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:32.765145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7409a6c3-0220-4cf4-9a0e-df2114eb8de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:32.784764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:32.865026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7409a6c3-0220-4cf4-9a0e-df2114eb8de4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:32.866926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7409a6c3-0220-4cf4-9a0e-df2114eb8de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:33.012885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 665/1365] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:33.066421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34048a12-efb8-4715-8ac0-d0063195d1fb None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:33.066994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34048a12-efb8-4715-8ac0-d0063195d1fb None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:33.067634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34048a12-efb8-4715-8ac0-d0063195d1fb None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:33.069175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 701/1366] 10.222.0.72 () {54 vars in 961 bytes} [Mon Jul 11 14:32:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:33.078600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5040b4-8bc5-4429-9dfd-095302685138 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:33.079151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5040b4-8bc5-4429-9dfd-095302685138 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:33.079646 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5040b4-8bc5-4429-9dfd-095302685138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:33.190915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1d5040b4-8bc5-4429-9dfd-095302685138 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:33.192368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1d5040b4-8bc5-4429-9dfd-095302685138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:33.234886 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:33.362473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 666/1367] 10.222.0.72 () {58 vars in 1058 bytes} [Mon Jul 11 14:32:33 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:33.363859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34048a12-efb8-4715-8ac0-d0063195d1fb None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:33.365444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-34048a12-efb8-4715-8ac0-d0063195d1fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:33.574277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34048a12-efb8-4715-8ac0-d0063195d1fb None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:33.601060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:33.740870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d346281-c08a-45c8-a0e0-2dd52c9b42c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:33.741714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d346281-c08a-45c8-a0e0-2dd52c9b42c0 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:33.742000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d346281-c08a-45c8-a0e0-2dd52c9b42c0 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:33.742321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d346281-c08a-45c8-a0e0-2dd52c9b42c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:33.744645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9d346281-c08a-45c8-a0e0-2dd52c9b42c0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:33.862241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d346281-c08a-45c8-a0e0-2dd52c9b42c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:33.863987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d346281-c08a-45c8-a0e0-2dd52c9b42c0 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:33.890482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 702/1368] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:34.892624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2753e2f-a9b8-4b85-bd64-26a2b405b067 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:34.893184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2753e2f-a9b8-4b85-bd64-26a2b405b067 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:34.893184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2753e2f-a9b8-4b85-bd64-26a2b405b067 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:34.900360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c8751c-6bd3-4ac0-a4f6-a911865dd8e5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:34.900855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c8751c-6bd3-4ac0-a4f6-a911865dd8e5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:34.900938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c8751c-6bd3-4ac0-a4f6-a911865dd8e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:34.906964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 667/1369] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:34.937362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:34.985249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-76c8751c-6bd3-4ac0-a4f6-a911865dd8e5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:34.986509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-76c8751c-6bd3-4ac0-a4f6-a911865dd8e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:35.133245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 703/1370] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:35.583894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3230f35e-9330-43d9-83f5-87b8184a62d9 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:35.584261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3230f35e-9330-43d9-83f5-87b8184a62d9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:35.584468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3230f35e-9330-43d9-83f5-87b8184a62d9 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:35.588071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 668/1371] 10.222.0.72 () {54 vars in 963 bytes} [Mon Jul 11 14:32:35 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:35.599703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0ab13-8be0-4a4b-b15c-5ff0e464ac47 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:35.600883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0ab13-8be0-4a4b-b15c-5ff0e464ac47 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:35.602097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0ab13-8be0-4a4b-b15c-5ff0e464ac47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:35.739545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c8d0ab13-8be0-4a4b-b15c-5ff0e464ac47 None None] MFA Rules not processed for user `4f1b0819d52d481898b354a5ae182746`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:35.741475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c8d0ab13-8be0-4a4b-b15c-5ff0e464ac47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:35.802125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:35.842897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 704/1372] 10.222.0.72 () {58 vars in 1060 bytes} [Mon Jul 11 14:32:35 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:35.848437 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3230f35e-9330-43d9-83f5-87b8184a62d9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:35.850288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3230f35e-9330-43d9-83f5-87b8184a62d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:35.972030 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3230f35e-9330-43d9-83f5-87b8184a62d9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:36.018814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:36.058524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:36.059101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:36.059380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:36.059654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:36.061517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:36.110100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e003ca1-79b6-4a7a-b305-be9dd3fd5b3c None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:36.110676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e003ca1-79b6-4a7a-b305-be9dd3fd5b3c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:36.110791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e003ca1-79b6-4a7a-b305-be9dd3fd5b3c None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:36.112245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 705/1373] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:32:36.117380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5104f1e-b0a3-45d3-9b71-924d72729bca None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:36.117677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5104f1e-b0a3-45d3-9b71-924d72729bca None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:36.117963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5104f1e-b0a3-45d3-9b71-924d72729bca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:36.191440 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b5104f1e-b0a3-45d3-9b71-924d72729bca None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:36.191758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:36.192831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b5104f1e-b0a3-45d3-9b71-924d72729bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:36.193229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:36.215691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 669/1374] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:36.323856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 706/1375] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:32:36.359657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:36.361230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:36.447292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:36.482095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81285e7-e9df-40d2-89e1-788eca4fa639 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:36.603577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23218ecf-5a94-4b51-9c0b-cc0f4bef41f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:36.604300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23218ecf-5a94-4b51-9c0b-cc0f4bef41f0 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:36.604584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23218ecf-5a94-4b51-9c0b-cc0f4bef41f0 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:36.604861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23218ecf-5a94-4b51-9c0b-cc0f4bef41f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:36.606550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-23218ecf-5a94-4b51-9c0b-cc0f4bef41f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:36.713871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:36.727475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23218ecf-5a94-4b51-9c0b-cc0f4bef41f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:36.729220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23218ecf-5a94-4b51-9c0b-cc0f4bef41f0 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:36.753170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 670/1376] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:38.842516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f2ca0e-0099-4146-a49a-495b3ccf091b None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:38.842516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f2ca0e-0099-4146-a49a-495b3ccf091b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:38.843178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f2ca0e-0099-4146-a49a-495b3ccf091b None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:38.844183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 707/1377] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:38.851777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd07cb3-f25e-45b9-a429-fb4b1736968c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:38.852645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd07cb3-f25e-45b9-a429-fb4b1736968c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:38.853108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd07cb3-f25e-45b9-a429-fb4b1736968c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:38.943151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-efd07cb3-f25e-45b9-a429-fb4b1736968c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:38.944189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-efd07cb3-f25e-45b9-a429-fb4b1736968c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:39.057978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 671/1378] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:39.104095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f2ca0e-0099-4146-a49a-495b3ccf091b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:39.105550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-42f2ca0e-0099-4146-a49a-495b3ccf091b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:39.170229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:39.283893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f2ca0e-0099-4146-a49a-495b3ccf091b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:39.419637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22d9fe0-ada5-4d36-9858-9e927619f482 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:39.420563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22d9fe0-ada5-4d36-9858-9e927619f482 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:39.420843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22d9fe0-ada5-4d36-9858-9e927619f482 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:39.421119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22d9fe0-ada5-4d36-9858-9e927619f482 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:39.422783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d22d9fe0-ada5-4d36-9858-9e927619f482 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:39.435190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:39.525072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22d9fe0-ada5-4d36-9858-9e927619f482 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:39.527708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22d9fe0-ada5-4d36-9858-9e927619f482 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:39.546018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 708/1379] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:41.663199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae25b03-cbbf-4cca-b028-bea8df608c9f None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:41.663778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae25b03-cbbf-4cca-b028-bea8df608c9f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:41.663778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae25b03-cbbf-4cca-b028-bea8df608c9f None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:41.665279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 672/1380] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:41.671615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f91402e-e80c-4e86-b7fc-829c1dc51030 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:41.672105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f91402e-e80c-4e86-b7fc-829c1dc51030 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:41.672208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f91402e-e80c-4e86-b7fc-829c1dc51030 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:41.743063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1f91402e-e80c-4e86-b7fc-829c1dc51030 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:41.744298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1f91402e-e80c-4e86-b7fc-829c1dc51030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:41.841678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:41.856576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 709/1381] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:41.900418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae25b03-cbbf-4cca-b028-bea8df608c9f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:41.902549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bae25b03-cbbf-4cca-b028-bea8df608c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:41.981583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:42.082648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae25b03-cbbf-4cca-b028-bea8df608c9f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:42.219315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1564213-77ec-467d-bbc9-4689d699d54f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:42.219969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1564213-77ec-467d-bbc9-4689d699d54f service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:42.220257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1564213-77ec-467d-bbc9-4689d699d54f service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:42.220535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1564213-77ec-467d-bbc9-4689d699d54f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:42.222306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d1564213-77ec-467d-bbc9-4689d699d54f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:42.251793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:42.357162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1564213-77ec-467d-bbc9-4689d699d54f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:42.359264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1564213-77ec-467d-bbc9-4689d699d54f service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:42.381849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 673/1382] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:44.735167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549a68f4-bc41-4be0-8fbe-447b502239c1 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:44.735805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549a68f4-bc41-4be0-8fbe-447b502239c1 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:44.735805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549a68f4-bc41-4be0-8fbe-447b502239c1 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:44.737104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 710/1383] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:44.742950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b78536-6b36-4168-ba12-9cd129f6ca24 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:44.743487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b78536-6b36-4168-ba12-9cd129f6ca24 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:44.743783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b78536-6b36-4168-ba12-9cd129f6ca24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:44.834138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-90b78536-6b36-4168-ba12-9cd129f6ca24 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:44.835864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-90b78536-6b36-4168-ba12-9cd129f6ca24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:44.965539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:44.970522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 674/1384] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:45.031830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549a68f4-bc41-4be0-8fbe-447b502239c1 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:45.033451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-549a68f4-bc41-4be0-8fbe-447b502239c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:45.236919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549a68f4-bc41-4be0-8fbe-447b502239c1 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:45.321056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:45.409032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12080c98-0b93-4bd7-956e-a78f3fdf46ed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:45.410056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12080c98-0b93-4bd7-956e-a78f3fdf46ed service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:45.410645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12080c98-0b93-4bd7-956e-a78f3fdf46ed service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:45.411124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12080c98-0b93-4bd7-956e-a78f3fdf46ed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:45.414639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-12080c98-0b93-4bd7-956e-a78f3fdf46ed service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:45.531887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12080c98-0b93-4bd7-956e-a78f3fdf46ed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:45.534862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12080c98-0b93-4bd7-956e-a78f3fdf46ed service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:45.571892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 711/1385] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 540 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:47.904961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742151a2-c311-41fb-ac81-1708dc572a36 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:47.905848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742151a2-c311-41fb-ac81-1708dc572a36 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:47.906245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742151a2-c311-41fb-ac81-1708dc572a36 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:47.909546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 675/1386] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:47.920254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23bbffe-1842-4c0b-8029-efa4f8fd4e06 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:47.920871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23bbffe-1842-4c0b-8029-efa4f8fd4e06 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:47.921094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23bbffe-1842-4c0b-8029-efa4f8fd4e06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:48.025453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d23bbffe-1842-4c0b-8029-efa4f8fd4e06 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:48.027787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d23bbffe-1842-4c0b-8029-efa4f8fd4e06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:48.074407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:48.203043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 712/1387] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:48.278257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742151a2-c311-41fb-ac81-1708dc572a36 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:48.280392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-742151a2-c311-41fb-ac81-1708dc572a36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:48.494323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742151a2-c311-41fb-ac81-1708dc572a36 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:48.596627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:48.666852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc2ac95-4c8c-4045-8ca8-6cb8b2034aac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:48.667332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc2ac95-4c8c-4045-8ca8-6cb8b2034aac service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:48.667632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc2ac95-4c8c-4045-8ca8-6cb8b2034aac service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:48.667926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc2ac95-4c8c-4045-8ca8-6cb8b2034aac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:48.670486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6bc2ac95-4c8c-4045-8ca8-6cb8b2034aac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:48.775462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc2ac95-4c8c-4045-8ca8-6cb8b2034aac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:48.777659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc2ac95-4c8c-4045-8ca8-6cb8b2034aac service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:48.804489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 676/1388] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:51.082021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7aa58-7a0c-47e6-85a5-51814e8166e8 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:51.082746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7aa58-7a0c-47e6-85a5-51814e8166e8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:51.082746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7aa58-7a0c-47e6-85a5-51814e8166e8 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:51.084221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 713/1389] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:51.090292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd0dca-277a-42bb-a5fb-2274fc32f9bf None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:51.090838 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd0dca-277a-42bb-a5fb-2274fc32f9bf None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:51.090957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd0dca-277a-42bb-a5fb-2274fc32f9bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:51.171670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d2bd0dca-277a-42bb-a5fb-2274fc32f9bf None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:51.172915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d2bd0dca-277a-42bb-a5fb-2274fc32f9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:51.215722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:51.294735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 677/1390] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:51.347340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7aa58-7a0c-47e6-85a5-51814e8166e8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:51.349281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-65e7aa58-7a0c-47e6-85a5-51814e8166e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:51.560838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7aa58-7a0c-47e6-85a5-51814e8166e8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:51.571804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:51.641952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca0c8b6-d18b-41ac-bc7b-977b9e8aed11 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:51.642333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca0c8b6-d18b-41ac-bc7b-977b9e8aed11 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:51.642525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca0c8b6-d18b-41ac-bc7b-977b9e8aed11 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:51.643848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 678/1391] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:32:51.649150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aaf655-098a-4b90-b1cc-832cc27c20c2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:51.649442 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aaf655-098a-4b90-b1cc-832cc27c20c2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:51.649728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aaf655-098a-4b90-b1cc-832cc27c20c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:51.709276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a50b23-6c81-4ad9-b298-9fa8a0389b30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:51.709944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a50b23-6c81-4ad9-b298-9fa8a0389b30 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:51.710364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a50b23-6c81-4ad9-b298-9fa8a0389b30 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:51.710673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a50b23-6c81-4ad9-b298-9fa8a0389b30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:51.712423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d8a50b23-6c81-4ad9-b298-9fa8a0389b30 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:51.724923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d0aaf655-098a-4b90-b1cc-832cc27c20c2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:51.726136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d0aaf655-098a-4b90-b1cc-832cc27c20c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:51.818020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a50b23-6c81-4ad9-b298-9fa8a0389b30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:51.819762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a50b23-6c81-4ad9-b298-9fa8a0389b30 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:51.835200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 679/1392] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:32:51.841333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 714/1393] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:51.894321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c4c54a-85b8-412d-b029-fe0c7c0abb1c None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:51.894548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c4c54a-85b8-412d-b029-fe0c7c0abb1c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:51.894799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c4c54a-85b8-412d-b029-fe0c7c0abb1c None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:51.896195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 680/1394] 10.222.0.72 () {54 vars in 962 bytes} [Mon Jul 11 14:32:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:51.901873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35482ad0-564b-428f-a6b4-87c93ed16dce None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:51.902167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35482ad0-564b-428f-a6b4-87c93ed16dce None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:51.902589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35482ad0-564b-428f-a6b4-87c93ed16dce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:51.918463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:52.001468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-35482ad0-564b-428f-a6b4-87c93ed16dce None None] MFA Rules not processed for user `fb6c2e2ddb3b4d488fa25fa8e258fec0`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:52.002824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-35482ad0-564b-428f-a6b4-87c93ed16dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:52.107826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 715/1395] 10.222.0.72 () {58 vars in 1059 bytes} [Mon Jul 11 14:32:51 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:52.115784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c4c54a-85b8-412d-b029-fe0c7c0abb1c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:52.117322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-22c4c54a-85b8-412d-b029-fe0c7c0abb1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:52.139282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:52.250271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c4c54a-85b8-412d-b029-fe0c7c0abb1c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:52.350931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c3c302-52f8-431f-91f7-428864b103d3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fb6c2e2ddb3b4d488fa25fa8e258fec0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:52.351605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c3c302-52f8-431f-91f7-428864b103d3 service glance] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:52.351887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c3c302-52f8-431f-91f7-428864b103d3 service glance] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:52.352166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c3c302-52f8-431f-91f7-428864b103d3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:52.354452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-59c3c302-52f8-431f-91f7-428864b103d3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:52.438726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:52.497038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c3c302-52f8-431f-91f7-428864b103d3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:52.499100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c3c302-52f8-431f-91f7-428864b103d3 service glance] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:52.523194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 681/1396] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:53.288226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a33e9-6336-41fc-92ae-9a0333153f68 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:53.288771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a33e9-6336-41fc-92ae-9a0333153f68 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:53.288975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a33e9-6336-41fc-92ae-9a0333153f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:53.364926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-544a33e9-6336-41fc-92ae-9a0333153f68 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:53.366147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-544a33e9-6336-41fc-92ae-9a0333153f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:53.431605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:53.472211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 716/1397] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:32:53 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:53.538010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320d238-674f-433c-a71e-d057209a85fd None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:53.538706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320d238-674f-433c-a71e-d057209a85fd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:53.538706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e320d238-674f-433c-a71e-d057209a85fd None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:53.542665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 682/1398] 10.222.0.72 () {54 vars in 942 bytes} [Mon Jul 11 14:32:53 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:53.552778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25819941-d23e-4f9e-ba89-a9ffe405a03a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:53.553068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25819941-d23e-4f9e-ba89-a9ffe405a03a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:53.553349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25819941-d23e-4f9e-ba89-a9ffe405a03a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:53.670751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-25819941-d23e-4f9e-ba89-a9ffe405a03a None None] MFA Rules not processed for user `055512e4e3744833ac6a9e61d22f0b63`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:53.672171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-25819941-d23e-4f9e-ba89-a9ffe405a03a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:53.810247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 717/1399] 10.222.0.72 () {58 vars in 1039 bytes} [Mon Jul 11 14:32:53 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:53.821125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320d238-674f-433c-a71e-d057209a85fd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:53.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e320d238-674f-433c-a71e-d057209a85fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:53.997846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e320d238-674f-433c-a71e-d057209a85fd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:54.000162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dad8fea-5ac9-4d0a-9892-14ad44e1b259 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:54.000457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dad8fea-5ac9-4d0a-9892-14ad44e1b259 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:54.000750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dad8fea-5ac9-4d0a-9892-14ad44e1b259 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:54.002309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 718/1400] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:32:54.007613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:54.007960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:54.008180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:54.045549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:54.053288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:54.092948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:54.094149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:54.139193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3962be49-6d36-413d-9fb1-e6a2bb53a137 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:54.139896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3962be49-6d36-413d-9fb1-e6a2bb53a137 service swift] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:54.140196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3962be49-6d36-413d-9fb1-e6a2bb53a137 service swift] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:54.140609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3962be49-6d36-413d-9fb1-e6a2bb53a137 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:54.142405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3962be49-6d36-413d-9fb1-e6a2bb53a137 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:54.228903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 719/1401] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:32:54.259316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3962be49-6d36-413d-9fb1-e6a2bb53a137 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:54.261508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3962be49-6d36-413d-9fb1-e6a2bb53a137 service swift] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:54.272657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 683/1402] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:32:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:32:54.279541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:54.280767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:54.446781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:54.452946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca417bc-e55f-4880-beaf-028d41f9ad6c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:54.630222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94661a76-0e99-4611-a174-520d6345fce4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:54.631210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94661a76-0e99-4611-a174-520d6345fce4 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:54.631623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94661a76-0e99-4611-a174-520d6345fce4 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:54.631914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94661a76-0e99-4611-a174-520d6345fce4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:54.633706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-94661a76-0e99-4611-a174-520d6345fce4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:54.771141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94661a76-0e99-4611-a174-520d6345fce4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:54.773842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94661a76-0e99-4611-a174-520d6345fce4 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:54.805575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 720/1403] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 526 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:56.993192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824fc30-cf21-46bd-9c77-0bb271459cd5 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:56.993752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824fc30-cf21-46bd-9c77-0bb271459cd5 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:56.993752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824fc30-cf21-46bd-9c77-0bb271459cd5 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:56.995996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 684/1404] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:57.001165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e54dbc2-24a5-46f6-9a2f-a480c8c15e1f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:57.001621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e54dbc2-24a5-46f6-9a2f-a480c8c15e1f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:57.001794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e54dbc2-24a5-46f6-9a2f-a480c8c15e1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:57.010319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:57.085452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9e54dbc2-24a5-46f6-9a2f-a480c8c15e1f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:57.086715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9e54dbc2-24a5-46f6-9a2f-a480c8c15e1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:57.196385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 721/1405] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:32:57.241786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824fc30-cf21-46bd-9c77-0bb271459cd5 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:32:57.243568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b824fc30-cf21-46bd-9c77-0bb271459cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:57.329402 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:57.419298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824fc30-cf21-46bd-9c77-0bb271459cd5 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:32:57.567544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380e49e2-3bf0-4e6f-999b-b5380be88eb3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:32:57.568308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e49e2-3bf0-4e6f-999b-b5380be88eb3 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:57.568664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e49e2-3bf0-4e6f-999b-b5380be88eb3 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:57.569005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e49e2-3bf0-4e6f-999b-b5380be88eb3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:57.570830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-380e49e2-3bf0-4e6f-999b-b5380be88eb3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:32:57.612239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:32:57.684523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380e49e2-3bf0-4e6f-999b-b5380be88eb3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:32:57.686128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380e49e2-3bf0-4e6f-999b-b5380be88eb3 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:32:57.708883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 685/1406] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:32:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:32:59.855073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d14732a-42eb-42f6-a7ba-ac3ac329d1a6 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:59.855697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d14732a-42eb-42f6-a7ba-ac3ac329d1a6 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:59.855697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d14732a-42eb-42f6-a7ba-ac3ac329d1a6 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:59.857641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 722/1407] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:32:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:32:59.863387 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a895ce30-8913-4ca4-afbe-59301871a25a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:32:59.863940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a895ce30-8913-4ca4-afbe-59301871a25a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:32:59.863940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a895ce30-8913-4ca4-afbe-59301871a25a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:32:59.942998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a895ce30-8913-4ca4-afbe-59301871a25a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:32:59.944128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a895ce30-8913-4ca4-afbe-59301871a25a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:00.052914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 686/1408] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:32:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:00.099914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d14732a-42eb-42f6-a7ba-ac3ac329d1a6 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:00.101544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1d14732a-42eb-42f6-a7ba-ac3ac329d1a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:00.212685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:00.304514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d14732a-42eb-42f6-a7ba-ac3ac329d1a6 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:00.499599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b100ccee-5275-461f-a437-89f03e46f9b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:00.500387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b100ccee-5275-461f-a437-89f03e46f9b9 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:00.500681 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b100ccee-5275-461f-a437-89f03e46f9b9 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:00.500955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b100ccee-5275-461f-a437-89f03e46f9b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:00.503350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b100ccee-5275-461f-a437-89f03e46f9b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:00.571994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:00.651928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b100ccee-5275-461f-a437-89f03e46f9b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:00.653940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b100ccee-5275-461f-a437-89f03e46f9b9 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:00.685219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 723/1409] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 586 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:10.050805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:10.680023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=26 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:17.986062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b536b-ffd5-4a6b-8b8a-93f62f50e522 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:17.986666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b536b-ffd5-4a6b-8b8a-93f62f50e522 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:17.987090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b536b-ffd5-4a6b-8b8a-93f62f50e522 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:17.989524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 687/1410] 10.222.0.72 () {54 vars in 918 bytes} [Mon Jul 11 14:33:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:17.996514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dc6438-fa8c-4d18-9029-f99b91c5f132 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:17.997069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dc6438-fa8c-4d18-9029-f99b91c5f132 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:17.997209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dc6438-fa8c-4d18-9029-f99b91c5f132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:18.085789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-00dc6438-fa8c-4d18-9029-f99b91c5f132 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:18.087793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-00dc6438-fa8c-4d18-9029-f99b91c5f132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:18.213710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 724/1411] 10.222.0.72 () {58 vars in 1015 bytes} [Mon Jul 11 14:33:17 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:18.230975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7286f5b-6281-45f4-8263-19282c6eba29 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:18.231285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7286f5b-6281-45f4-8263-19282c6eba29 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:18.231549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7286f5b-6281-45f4-8263-19282c6eba29 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:18.233029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 688/1412] 10.222.0.72 () {54 vars in 918 bytes} [Mon Jul 11 14:33:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:18.238419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c5763e-1fa2-4765-8f75-4af1251673e9 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:18.238825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c5763e-1fa2-4765-8f75-4af1251673e9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:18.239205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c5763e-1fa2-4765-8f75-4af1251673e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:18.335490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:18.339640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-84c5763e-1fa2-4765-8f75-4af1251673e9 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:18.341543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-84c5763e-1fa2-4765-8f75-4af1251673e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:18.628011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 725/1413] 10.222.0.72 () {58 vars in 1015 bytes} [Mon Jul 11 14:33:18 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 391 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:21.336136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a5c976-0440-48de-a13a-e009e75e4fad None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:21.336741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a5c976-0440-48de-a13a-e009e75e4fad None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:21.336847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a5c976-0440-48de-a13a-e009e75e4fad None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:21.338482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 689/1414] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:21.344545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33207562-b06e-4541-b15c-ff8793bcda19 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:21.345015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33207562-b06e-4541-b15c-ff8793bcda19 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:21.345111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33207562-b06e-4541-b15c-ff8793bcda19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:21.419082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-33207562-b06e-4541-b15c-ff8793bcda19 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:21.420317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-33207562-b06e-4541-b15c-ff8793bcda19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:21.483562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:21.531426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 726/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:21.583971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a5c976-0440-48de-a13a-e009e75e4fad None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:21.586255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-31a5c976-0440-48de-a13a-e009e75e4fad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:21.754625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a5c976-0440-48de-a13a-e009e75e4fad None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:21.832865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:21.876165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1293ce2c-2da3-444e-8aa0-d910535ea73a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:21.876740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1293ce2c-2da3-444e-8aa0-d910535ea73a service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:21.877031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1293ce2c-2da3-444e-8aa0-d910535ea73a service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:21.877311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1293ce2c-2da3-444e-8aa0-d910535ea73a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:21.879361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1293ce2c-2da3-444e-8aa0-d910535ea73a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:21.979055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1293ce2c-2da3-444e-8aa0-d910535ea73a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:21.980678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1293ce2c-2da3-444e-8aa0-d910535ea73a service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:22.001097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 690/1416] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:22.945771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6487e7-fcc0-438a-8227-32748a7e5f56 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:22.946457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6487e7-fcc0-438a-8227-32748a7e5f56 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:22.946457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6487e7-fcc0-438a-8227-32748a7e5f56 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:22.948273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 727/1417] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:22.955078 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169be9bb-17e2-43a5-940a-e2b5d589516c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:22.955539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169be9bb-17e2-43a5-940a-e2b5d589516c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:22.955659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169be9bb-17e2-43a5-940a-e2b5d589516c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:23.041105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-169be9bb-17e2-43a5-940a-e2b5d589516c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:23.042791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-169be9bb-17e2-43a5-940a-e2b5d589516c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:23.089829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:23.172360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 691/1418] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:23.342639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6487e7-fcc0-438a-8227-32748a7e5f56 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:23.345081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3c6487e7-fcc0-438a-8227-32748a7e5f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:23.510950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6487e7-fcc0-438a-8227-32748a7e5f56 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:23.558522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:23.662358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6fef2b-a4ce-488f-bc46-d18f1c2b3db2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:23.662980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6fef2b-a4ce-488f-bc46-d18f1c2b3db2 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:23.663270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6fef2b-a4ce-488f-bc46-d18f1c2b3db2 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:23.663559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6fef2b-a4ce-488f-bc46-d18f1c2b3db2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:23.665289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5c6fef2b-a4ce-488f-bc46-d18f1c2b3db2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:23.765643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6fef2b-a4ce-488f-bc46-d18f1c2b3db2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:23.767330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6fef2b-a4ce-488f-bc46-d18f1c2b3db2 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:23.787970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 728/1419] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:24.970127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed52d05d-62a0-4a35-9273-259825d38d63 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:24.970750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed52d05d-62a0-4a35-9273-259825d38d63 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:24.970750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed52d05d-62a0-4a35-9273-259825d38d63 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:24.972144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 692/1420] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:24.979021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef62c25f-ba40-4b64-82e5-bfdab854d169 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:24.979821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef62c25f-ba40-4b64-82e5-bfdab854d169 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:24.979821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef62c25f-ba40-4b64-82e5-bfdab854d169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:25.084530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:25.123116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ef62c25f-ba40-4b64-82e5-bfdab854d169 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:25.125005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ef62c25f-ba40-4b64-82e5-bfdab854d169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:25.270899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 729/1421] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:33:25.459091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed52d05d-62a0-4a35-9273-259825d38d63 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:25.460693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ed52d05d-62a0-4a35-9273-259825d38d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:25.661090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed52d05d-62a0-4a35-9273-259825d38d63 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:25.678419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:25.802848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1585dda9-71a0-433f-a60d-39a1f1cab1f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:25.803528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1585dda9-71a0-433f-a60d-39a1f1cab1f3 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:25.803823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1585dda9-71a0-433f-a60d-39a1f1cab1f3 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:25.804095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1585dda9-71a0-433f-a60d-39a1f1cab1f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:25.805852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1585dda9-71a0-433f-a60d-39a1f1cab1f3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:25.927621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1585dda9-71a0-433f-a60d-39a1f1cab1f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:25.929161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1585dda9-71a0-433f-a60d-39a1f1cab1f3 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:25.955517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 693/1422] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:26.908848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe210b-4c7e-432c-b08b-f5dee1e5bd02 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:26.909425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe210b-4c7e-432c-b08b-f5dee1e5bd02 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:26.909425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fe210b-4c7e-432c-b08b-f5dee1e5bd02 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:26.910884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 730/1423] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:26.921128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e747a0-a7dc-49cc-bc66-c28198a02b3b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:26.921703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e747a0-a7dc-49cc-bc66-c28198a02b3b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:26.921979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e747a0-a7dc-49cc-bc66-c28198a02b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:26.953427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:27.023985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a4e747a0-a7dc-49cc-bc66-c28198a02b3b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:27.026215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a4e747a0-a7dc-49cc-bc66-c28198a02b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:27.158918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 694/1424] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:27.395455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe210b-4c7e-432c-b08b-f5dee1e5bd02 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:27.397499 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b6fe210b-4c7e-432c-b08b-f5dee1e5bd02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:27.529179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:27.572709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fe210b-4c7e-432c-b08b-f5dee1e5bd02 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:27.728084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6606a66-26fc-4b66-b94d-d32b7e501076 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:27.728747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6606a66-26fc-4b66-b94d-d32b7e501076 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:27.729033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6606a66-26fc-4b66-b94d-d32b7e501076 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:27.729307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6606a66-26fc-4b66-b94d-d32b7e501076 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:27.731074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d6606a66-26fc-4b66-b94d-d32b7e501076 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:27.823882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:27.842414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6606a66-26fc-4b66-b94d-d32b7e501076 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:27.844135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6606a66-26fc-4b66-b94d-d32b7e501076 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:27.868677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 731/1425] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:28.802780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8027a31d-fdaf-4a20-ac72-70c29a9e2bac None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:28.803551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8027a31d-fdaf-4a20-ac72-70c29a9e2bac None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:28.803551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8027a31d-fdaf-4a20-ac72-70c29a9e2bac None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:28.805277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 695/1426] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:28.811190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bffc41-bec5-409c-b224-02a5585e8e44 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:28.811700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bffc41-bec5-409c-b224-02a5585e8e44 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:28.811817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bffc41-bec5-409c-b224-02a5585e8e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:28.929156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-48bffc41-bec5-409c-b224-02a5585e8e44 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:28.931383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-48bffc41-bec5-409c-b224-02a5585e8e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:29.064897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 732/1427] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:33:29.226287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8027a31d-fdaf-4a20-ac72-70c29a9e2bac None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:29.228174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8027a31d-fdaf-4a20-ac72-70c29a9e2bac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:29.341599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:29.421162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8027a31d-fdaf-4a20-ac72-70c29a9e2bac None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:29.595846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5e31c3-1518-4e00-a610-1ef98d46c813 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:29.596454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5e31c3-1518-4e00-a610-1ef98d46c813 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:29.596745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5e31c3-1518-4e00-a610-1ef98d46c813 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:29.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5e31c3-1518-4e00-a610-1ef98d46c813 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:29.599000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3b5e31c3-1518-4e00-a610-1ef98d46c813 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:29.656678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:29.708267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5e31c3-1518-4e00-a610-1ef98d46c813 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:29.709929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5e31c3-1518-4e00-a610-1ef98d46c813 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:29.733059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 696/1428] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:30.569810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b2008-458c-4dee-a491-7fc929ee241a None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:30.570397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b2008-458c-4dee-a491-7fc929ee241a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:30.570397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b2008-458c-4dee-a491-7fc929ee241a None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:30.572386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 733/1429] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:30.578215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45153ef-4644-4f03-aaec-eaaf77d9d6bb None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:30.578727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45153ef-4644-4f03-aaec-eaaf77d9d6bb None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:30.578842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45153ef-4644-4f03-aaec-eaaf77d9d6bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:30.677085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c45153ef-4644-4f03-aaec-eaaf77d9d6bb None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:30.678398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c45153ef-4644-4f03-aaec-eaaf77d9d6bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:30.818499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 697/1430] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:30.985104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b2008-458c-4dee-a491-7fc929ee241a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:30.986707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a91b2008-458c-4dee-a491-7fc929ee241a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:31.042813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:31.155217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b2008-458c-4dee-a491-7fc929ee241a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:31.286758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6b574b-ead7-40cd-b3d2-6a8ee44fdd57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:31.287413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b574b-ead7-40cd-b3d2-6a8ee44fdd57 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:31.287695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b574b-ead7-40cd-b3d2-6a8ee44fdd57 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:31.287965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b574b-ead7-40cd-b3d2-6a8ee44fdd57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:31.289653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8e6b574b-ead7-40cd-b3d2-6a8ee44fdd57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:31.300854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:31.402260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6b574b-ead7-40cd-b3d2-6a8ee44fdd57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:31.404202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6b574b-ead7-40cd-b3d2-6a8ee44fdd57 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:31.426445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 734/1431] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:32.308942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25795481-b7a2-4348-bb59-a7c1e1bcfab7 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:32.309658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25795481-b7a2-4348-bb59-a7c1e1bcfab7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:32.309658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25795481-b7a2-4348-bb59-a7c1e1bcfab7 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:32.311019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 698/1432] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:32.316796 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49545c6-2701-488b-9d64-7cb4a5a93abc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:32.317435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49545c6-2701-488b-9d64-7cb4a5a93abc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:32.317548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49545c6-2701-488b-9d64-7cb4a5a93abc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:32.407435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e49545c6-2701-488b-9d64-7cb4a5a93abc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:32.408666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e49545c6-2701-488b-9d64-7cb4a5a93abc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:32.484722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:32.517391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 735/1433] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:32.691194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25795481-b7a2-4348-bb59-a7c1e1bcfab7 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:32.693142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-25795481-b7a2-4348-bb59-a7c1e1bcfab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:32.717655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:32.877564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25795481-b7a2-4348-bb59-a7c1e1bcfab7 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:32.978409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:33.008107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c3425b-a576-45a1-88a0-a41d1cbb9667 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:33.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c3425b-a576-45a1-88a0-a41d1cbb9667 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:33.009099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c3425b-a576-45a1-88a0-a41d1cbb9667 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:33.009388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c3425b-a576-45a1-88a0-a41d1cbb9667 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:33.011174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-75c3425b-a576-45a1-88a0-a41d1cbb9667 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:33.120104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c3425b-a576-45a1-88a0-a41d1cbb9667 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:33.122726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c3425b-a576-45a1-88a0-a41d1cbb9667 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:33.148166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 699/1434] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:34.007991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd365dcd-1e65-461b-bec1-91fba6a0ef18 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:34.008551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd365dcd-1e65-461b-bec1-91fba6a0ef18 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:34.008551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd365dcd-1e65-461b-bec1-91fba6a0ef18 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:34.010489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 736/1435] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:34.017599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77a71b0-86fd-4d5c-b779-c6d58508f727 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:34.018169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77a71b0-86fd-4d5c-b779-c6d58508f727 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:34.019132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77a71b0-86fd-4d5c-b779-c6d58508f727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:34.104025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d77a71b0-86fd-4d5c-b779-c6d58508f727 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:34.105332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d77a71b0-86fd-4d5c-b779-c6d58508f727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:34.153694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:34.214283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 700/1436] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:34.430998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd365dcd-1e65-461b-bec1-91fba6a0ef18 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:34.432649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cd365dcd-1e65-461b-bec1-91fba6a0ef18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:34.620793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd365dcd-1e65-461b-bec1-91fba6a0ef18 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:34.688750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:34.759194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2438ff8c-164a-4d35-8cf2-13f60061fc05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:34.759947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2438ff8c-164a-4d35-8cf2-13f60061fc05 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:34.760232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2438ff8c-164a-4d35-8cf2-13f60061fc05 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:34.760538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2438ff8c-164a-4d35-8cf2-13f60061fc05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:34.763080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2438ff8c-164a-4d35-8cf2-13f60061fc05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:34.871423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2438ff8c-164a-4d35-8cf2-13f60061fc05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:34.873152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2438ff8c-164a-4d35-8cf2-13f60061fc05 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:34.892508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 737/1437] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:33:35.765700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21ceb98-2eb5-4352-a798-fcd257d54948 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:35.766615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21ceb98-2eb5-4352-a798-fcd257d54948 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:35.766780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21ceb98-2eb5-4352-a798-fcd257d54948 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:35.768226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 701/1438] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:33:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:35.774849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e597aeed-3bd5-4386-b7f0-b284d9aaa1bc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:35.775409 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e597aeed-3bd5-4386-b7f0-b284d9aaa1bc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:35.775603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e597aeed-3bd5-4386-b7f0-b284d9aaa1bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:35.868690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e597aeed-3bd5-4386-b7f0-b284d9aaa1bc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:35.870196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e597aeed-3bd5-4386-b7f0-b284d9aaa1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:35.872680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:35.982772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 738/1439] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:33:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:36.147959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21ceb98-2eb5-4352-a798-fcd257d54948 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:36.149558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b21ceb98-2eb5-4352-a798-fcd257d54948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:36.332517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21ceb98-2eb5-4352-a798-fcd257d54948 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:36.369967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:36.481224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214b9a29-1bf5-48e6-ad76-0edb626b8dc4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:36.481791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b9a29-1bf5-48e6-ad76-0edb626b8dc4 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:36.482079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b9a29-1bf5-48e6-ad76-0edb626b8dc4 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:36.482356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b9a29-1bf5-48e6-ad76-0edb626b8dc4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:36.484180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-214b9a29-1bf5-48e6-ad76-0edb626b8dc4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:36.598995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214b9a29-1bf5-48e6-ad76-0edb626b8dc4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:36.600581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214b9a29-1bf5-48e6-ad76-0edb626b8dc4 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:36.622297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 702/1440] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:38.608921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0a4ef-17ca-4d60-a3d2-a2de2de4a6dd None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:38.609523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0a4ef-17ca-4d60-a3d2-a2de2de4a6dd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:38.609523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0a4ef-17ca-4d60-a3d2-a2de2de4a6dd None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:38.611026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 739/1441] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:38.616247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd05b33-7ed7-4dba-84cc-c6e2c77298a9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:38.616696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd05b33-7ed7-4dba-84cc-c6e2c77298a9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:38.616808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd05b33-7ed7-4dba-84cc-c6e2c77298a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:38.660233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:38.691920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6bd05b33-7ed7-4dba-84cc-c6e2c77298a9 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:38.693266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6bd05b33-7ed7-4dba-84cc-c6e2c77298a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:38.805924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 703/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:38.846299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af0a4ef-17ca-4d60-a3d2-a2de2de4a6dd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:38.847987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2af0a4ef-17ca-4d60-a3d2-a2de2de4a6dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:38.952795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af0a4ef-17ca-4d60-a3d2-a2de2de4a6dd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:38.986608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:39.031255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883306cf-25c5-412a-86bc-b9aca61ebc02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:39.031821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883306cf-25c5-412a-86bc-b9aca61ebc02 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:39.032103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883306cf-25c5-412a-86bc-b9aca61ebc02 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:39.032375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883306cf-25c5-412a-86bc-b9aca61ebc02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:39.034037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-883306cf-25c5-412a-86bc-b9aca61ebc02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:39.153309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883306cf-25c5-412a-86bc-b9aca61ebc02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:39.154801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883306cf-25c5-412a-86bc-b9aca61ebc02 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:39.175108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 740/1443] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:41.380378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc7fb1-246c-497e-96cd-fb609183bba0 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:41.380965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc7fb1-246c-497e-96cd-fb609183bba0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:41.380965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc7fb1-246c-497e-96cd-fb609183bba0 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:41.382317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 704/1444] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:41.387719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0458449-462f-41d6-ae0a-448934a582d1 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:41.388128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0458449-462f-41d6-ae0a-448934a582d1 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:41.388270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0458449-462f-41d6-ae0a-448934a582d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:41.454691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d0458449-462f-41d6-ae0a-448934a582d1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:41.455920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d0458449-462f-41d6-ae0a-448934a582d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:41.469581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:41.555463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 741/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:41.596611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc7fb1-246c-497e-96cd-fb609183bba0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:41.598185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e3bc7fb1-246c-497e-96cd-fb609183bba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:41.721375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc7fb1-246c-497e-96cd-fb609183bba0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:41.737728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:41.816131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129f1db0-7331-47b2-bce5-0b06342e3a02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:41.816796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f1db0-7331-47b2-bce5-0b06342e3a02 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:41.817079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f1db0-7331-47b2-bce5-0b06342e3a02 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:41.817356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f1db0-7331-47b2-bce5-0b06342e3a02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:41.819090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-129f1db0-7331-47b2-bce5-0b06342e3a02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:41.962588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129f1db0-7331-47b2-bce5-0b06342e3a02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:41.964825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129f1db0-7331-47b2-bce5-0b06342e3a02 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:42.000741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 705/1446] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:44.610971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838da6ff-56a0-4925-8d9d-2c6e9f2acadd None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:44.611502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838da6ff-56a0-4925-8d9d-2c6e9f2acadd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:44.611502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838da6ff-56a0-4925-8d9d-2c6e9f2acadd None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:44.612969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 742/1447] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:44.618972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeef1172-bea2-445d-b389-bc42f1d7b877 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:44.618972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeef1172-bea2-445d-b389-bc42f1d7b877 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:44.618972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeef1172-bea2-445d-b389-bc42f1d7b877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:44.682776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:44.712517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-eeef1172-bea2-445d-b389-bc42f1d7b877 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:44.714976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-eeef1172-bea2-445d-b389-bc42f1d7b877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:44.826120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 706/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:44.866777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838da6ff-56a0-4925-8d9d-2c6e9f2acadd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:44.869062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-838da6ff-56a0-4925-8d9d-2c6e9f2acadd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:44.979037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838da6ff-56a0-4925-8d9d-2c6e9f2acadd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:45.025260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:45.062078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b20b00c-8d1e-4999-9426-07607e2d2fdc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:45.062797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20b00c-8d1e-4999-9426-07607e2d2fdc service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:45.063256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20b00c-8d1e-4999-9426-07607e2d2fdc service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:45.063526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20b00c-8d1e-4999-9426-07607e2d2fdc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:45.065456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5b20b00c-8d1e-4999-9426-07607e2d2fdc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:45.212549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b20b00c-8d1e-4999-9426-07607e2d2fdc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:45.213941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b20b00c-8d1e-4999-9426-07607e2d2fdc service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:45.233548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 743/1449] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:47.514915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2780b615-ad15-4dfd-b7e9-67e134777526 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:47.515793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2780b615-ad15-4dfd-b7e9-67e134777526 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:47.515793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2780b615-ad15-4dfd-b7e9-67e134777526 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:47.516924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 707/1450] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:47.523286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1b693-841d-4dcd-b1a3-41cab0b055b5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:47.523890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1b693-841d-4dcd-b1a3-41cab0b055b5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:47.523890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1b693-841d-4dcd-b1a3-41cab0b055b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:47.603334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-0fb1b693-841d-4dcd-b1a3-41cab0b055b5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:47.604567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0fb1b693-841d-4dcd-b1a3-41cab0b055b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:47.627069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:47.712791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 744/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:47.756505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2780b615-ad15-4dfd-b7e9-67e134777526 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:47.758105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2780b615-ad15-4dfd-b7e9-67e134777526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:47.871641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2780b615-ad15-4dfd-b7e9-67e134777526 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:47.893277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:47.950952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466d7d1f-a048-4418-a03a-38b3a60d38a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:47.951620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466d7d1f-a048-4418-a03a-38b3a60d38a8 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:47.951914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466d7d1f-a048-4418-a03a-38b3a60d38a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:47.952205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466d7d1f-a048-4418-a03a-38b3a60d38a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:47.954201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-466d7d1f-a048-4418-a03a-38b3a60d38a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:48.072039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466d7d1f-a048-4418-a03a-38b3a60d38a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:48.073539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466d7d1f-a048-4418-a03a-38b3a60d38a8 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:48.093512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 708/1452] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:50.239354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cf55f2-1def-4a97-a798-2f2f57bedd10 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:50.239891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cf55f2-1def-4a97-a798-2f2f57bedd10 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:50.239891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cf55f2-1def-4a97-a798-2f2f57bedd10 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:50.241295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 745/1453] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:50.247357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fdf458-00c3-41a1-a0f4-db39f50e6de3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:50.247887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fdf458-00c3-41a1-a0f4-db39f50e6de3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:50.247996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fdf458-00c3-41a1-a0f4-db39f50e6de3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:50.318497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:50.322218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-08fdf458-00c3-41a1-a0f4-db39f50e6de3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:50.323685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-08fdf458-00c3-41a1-a0f4-db39f50e6de3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:50.428768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 709/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:50.467054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cf55f2-1def-4a97-a798-2f2f57bedd10 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:50.468605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-68cf55f2-1def-4a97-a798-2f2f57bedd10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:50.583117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cf55f2-1def-4a97-a798-2f2f57bedd10 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:50.634018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:50.661563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc7bcd-8a88-420d-a3bb-071be4324a00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:50.662795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc7bcd-8a88-420d-a3bb-071be4324a00 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:50.662795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc7bcd-8a88-420d-a3bb-071be4324a00 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:50.662795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc7bcd-8a88-420d-a3bb-071be4324a00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:50.664512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b4cc7bcd-8a88-420d-a3bb-071be4324a00 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:50.783437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc7bcd-8a88-420d-a3bb-071be4324a00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:50.784816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc7bcd-8a88-420d-a3bb-071be4324a00 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:50.804813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 746/1455] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:53.055772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1902d573-ecb1-4d65-b5cd-897dfcbae886 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:53.056608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1902d573-ecb1-4d65-b5cd-897dfcbae886 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:53.056608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1902d573-ecb1-4d65-b5cd-897dfcbae886 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:53.057766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 710/1456] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:53.065443 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a343a0-199f-4131-a488-cfc20287d64d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:53.065975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a343a0-199f-4131-a488-cfc20287d64d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:53.066076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a343a0-199f-4131-a488-cfc20287d64d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:53.137448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b3a343a0-199f-4131-a488-cfc20287d64d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:53.138740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b3a343a0-199f-4131-a488-cfc20287d64d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:53.169111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:53.243944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 747/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:53.281826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1902d573-ecb1-4d65-b5cd-897dfcbae886 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:53.283415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1902d573-ecb1-4d65-b5cd-897dfcbae886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:53.399188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1902d573-ecb1-4d65-b5cd-897dfcbae886 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:53.430630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:53.480510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4431a8-907e-414e-a898-bb2afebc26c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:53.481477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4431a8-907e-414e-a898-bb2afebc26c2 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:53.481775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4431a8-907e-414e-a898-bb2afebc26c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:53.482066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4431a8-907e-414e-a898-bb2afebc26c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:53.483823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-af4431a8-907e-414e-a898-bb2afebc26c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:53.604793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4431a8-907e-414e-a898-bb2afebc26c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:53.606252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4431a8-907e-414e-a898-bb2afebc26c2 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:53.626523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 711/1458] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:55.750691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48036a25-13d0-47e8-81bb-0de76fc9769e None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:55.751233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48036a25-13d0-47e8-81bb-0de76fc9769e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:55.751233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48036a25-13d0-47e8-81bb-0de76fc9769e None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:55.752658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 748/1459] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:55.758486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c3851-d7e7-4508-a25d-4bf8740c4118 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:55.758966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c3851-d7e7-4508-a25d-4bf8740c4118 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:55.759082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c3851-d7e7-4508-a25d-4bf8740c4118 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:55.826658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-603c3851-d7e7-4508-a25d-4bf8740c4118 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:55.828147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-603c3851-d7e7-4508-a25d-4bf8740c4118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:55.837320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:55.929053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 712/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:55.966742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48036a25-13d0-47e8-81bb-0de76fc9769e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:55.968297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-48036a25-13d0-47e8-81bb-0de76fc9769e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:56.082862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48036a25-13d0-47e8-81bb-0de76fc9769e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:33:56.140006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:56.163070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafb3785-2992-4735-be1f-5185051dafb5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:33:56.163720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb3785-2992-4735-be1f-5185051dafb5 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:56.164005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb3785-2992-4735-be1f-5185051dafb5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:56.164838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb3785-2992-4735-be1f-5185051dafb5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:56.166563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bafb3785-2992-4735-be1f-5185051dafb5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:56.279435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafb3785-2992-4735-be1f-5185051dafb5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:33:56.280907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafb3785-2992-4735-be1f-5185051dafb5 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:33:56.301729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 749/1461] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:33:59.733167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc309d99-50cb-47ea-9010-5ca83e0a2799 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:59.733752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc309d99-50cb-47ea-9010-5ca83e0a2799 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:59.733752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc309d99-50cb-47ea-9010-5ca83e0a2799 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:59.735157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 713/1462] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:33:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:33:59.743851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6657f6-11e1-4301-aab4-ef7a2ba9493c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:33:59.744357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6657f6-11e1-4301-aab4-ef7a2ba9493c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:33:59.744476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6657f6-11e1-4301-aab4-ef7a2ba9493c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:33:59.815121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-8b6657f6-11e1-4301-aab4-ef7a2ba9493c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:33:59.816348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8b6657f6-11e1-4301-aab4-ef7a2ba9493c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:33:59.858776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:33:59.921867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 750/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:33:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:33:59.961527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc309d99-50cb-47ea-9010-5ca83e0a2799 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:33:59.963155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dc309d99-50cb-47ea-9010-5ca83e0a2799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:00.078213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc309d99-50cb-47ea-9010-5ca83e0a2799 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:00.122112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:00.162159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b67ddc-bf0b-4b15-8012-40f0cd0caebc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:00.162853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b67ddc-bf0b-4b15-8012-40f0cd0caebc service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:00.163144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b67ddc-bf0b-4b15-8012-40f0cd0caebc service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:00.163427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b67ddc-bf0b-4b15-8012-40f0cd0caebc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:00.165704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-09b67ddc-bf0b-4b15-8012-40f0cd0caebc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:00.290704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b67ddc-bf0b-4b15-8012-40f0cd0caebc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:00.292207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b67ddc-bf0b-4b15-8012-40f0cd0caebc service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:00.313640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 714/1464] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:33:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:02.508846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa0cd7-6a0f-4f6c-bb30-7133d26c0369 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:02.509412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa0cd7-6a0f-4f6c-bb30-7133d26c0369 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:02.509412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa0cd7-6a0f-4f6c-bb30-7133d26c0369 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:02.510903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 751/1465] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:34:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:02.517480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d89a5-7d45-46ad-bcff-a35357bf0176 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:02.518154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d89a5-7d45-46ad-bcff-a35357bf0176 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:02.518154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d89a5-7d45-46ad-bcff-a35357bf0176 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:02.622418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-577d89a5-7d45-46ad-bcff-a35357bf0176 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:02.630261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-577d89a5-7d45-46ad-bcff-a35357bf0176 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:02.660349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:02.770035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 715/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:34:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:02.814744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caa0cd7-6a0f-4f6c-bb30-7133d26c0369 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:02.817058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7caa0cd7-6a0f-4f6c-bb30-7133d26c0369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:02.976249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caa0cd7-6a0f-4f6c-bb30-7133d26c0369 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:03.057839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:03.068037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2878a7c-a038-44f5-9c9d-f8d7795208d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:03.069056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2878a7c-a038-44f5-9c9d-f8d7795208d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:03.069218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2878a7c-a038-44f5-9c9d-f8d7795208d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:03.069682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2878a7c-a038-44f5-9c9d-f8d7795208d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:03.072474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c2878a7c-a038-44f5-9c9d-f8d7795208d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:03.223981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2878a7c-a038-44f5-9c9d-f8d7795208d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:03.225456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2878a7c-a038-44f5-9c9d-f8d7795208d2 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:03.267066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 752/1467] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:05.422722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ff66e-d74c-4ea9-a381-14518cf7765c None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:05.423403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ff66e-d74c-4ea9-a381-14518cf7765c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:05.423403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ff66e-d74c-4ea9-a381-14518cf7765c None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:05.426046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 716/1468] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:34:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:05.431425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee748945-539e-4578-88dc-8be4f68dd6a0 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:05.431961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee748945-539e-4578-88dc-8be4f68dd6a0 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:05.431961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee748945-539e-4578-88dc-8be4f68dd6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:05.526679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ee748945-539e-4578-88dc-8be4f68dd6a0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:05.527932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ee748945-539e-4578-88dc-8be4f68dd6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:05.582916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:05.647074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 753/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:34:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:05.687623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ff66e-d74c-4ea9-a381-14518cf7765c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:05.689554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3a2ff66e-d74c-4ea9-a381-14518cf7765c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:05.838760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ff66e-d74c-4ea9-a381-14518cf7765c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:05.912812 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:05.949834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b1a56a-0117-4fb0-81b0-1b177d71d947 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:05.950415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b1a56a-0117-4fb0-81b0-1b177d71d947 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:05.950734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b1a56a-0117-4fb0-81b0-1b177d71d947 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:05.951015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b1a56a-0117-4fb0-81b0-1b177d71d947 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:05.953047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-30b1a56a-0117-4fb0-81b0-1b177d71d947 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:06.133886 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b1a56a-0117-4fb0-81b0-1b177d71d947 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:06.136621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b1a56a-0117-4fb0-81b0-1b177d71d947 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:06.171743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 717/1470] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:15.644365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=22 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:16.168600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=66 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:26.704846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c915ce-3c93-41b0-9995-ef252180dfe4 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:26.705481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c915ce-3c93-41b0-9995-ef252180dfe4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:26.705644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c915ce-3c93-41b0-9995-ef252180dfe4 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:26.708723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 754/1471] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:34:26 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:34:26.715747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7616a0-334e-4a27-a874-eff340317f84 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:26.716253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7616a0-334e-4a27-a874-eff340317f84 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:26.716372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7616a0-334e-4a27-a874-eff340317f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:26.796979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ad7616a0-334e-4a27-a874-eff340317f84 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:26.798131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ad7616a0-334e-4a27-a874-eff340317f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:26.923323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 718/1472] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:34:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:27.035656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b97921-7050-4ce0-826a-8adb14f9a573 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:27.035921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b97921-7050-4ce0-826a-8adb14f9a573 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:27.036185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b97921-7050-4ce0-826a-8adb14f9a573 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:27.037562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 755/1473] 10.222.0.72 () {54 vars in 962 bytes} [Mon Jul 11 14:34:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:27.046400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a120759e-0527-4652-a981-9c54c06646d8 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:27.046400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a120759e-0527-4652-a981-9c54c06646d8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:27.047157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a120759e-0527-4652-a981-9c54c06646d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:27.141982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a120759e-0527-4652-a981-9c54c06646d8 None None] MFA Rules not processed for user `fb6c2e2ddb3b4d488fa25fa8e258fec0`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:27.143680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a120759e-0527-4652-a981-9c54c06646d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:27.172405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:27.246649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 719/1474] 10.222.0.72 () {58 vars in 1059 bytes} [Mon Jul 11 14:34:27 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:27.254899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b97921-7050-4ce0-826a-8adb14f9a573 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:27.256536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f5b97921-7050-4ce0-826a-8adb14f9a573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:27.401853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b97921-7050-4ce0-826a-8adb14f9a573 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:27.497177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65006ffc-691b-4663-a2b3-57ac416c1362 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fb6c2e2ddb3b4d488fa25fa8e258fec0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:27.498232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65006ffc-691b-4663-a2b3-57ac416c1362 service glance] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:27.499100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65006ffc-691b-4663-a2b3-57ac416c1362 service glance] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:27.499251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65006ffc-691b-4663-a2b3-57ac416c1362 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:27.502335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-65006ffc-691b-4663-a2b3-57ac416c1362 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:27.579945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:27.655515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65006ffc-691b-4663-a2b3-57ac416c1362 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:27.657341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65006ffc-691b-4663-a2b3-57ac416c1362 service glance] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:27.685056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 756/1475] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:29.939334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4382e3be-f3d7-47b1-ae41-e599d75f2f41 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:29.939865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4382e3be-f3d7-47b1-ae41-e599d75f2f41 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:29.939865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4382e3be-f3d7-47b1-ae41-e599d75f2f41 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:29.941245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 720/1476] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:34:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:29.946734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6be27-fa9b-4a94-a518-1c8377b8de07 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:29.947290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6be27-fa9b-4a94-a518-1c8377b8de07 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:29.947290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6be27-fa9b-4a94-a518-1c8377b8de07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:30.026302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e5b6be27-fa9b-4a94-a518-1c8377b8de07 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:30.028367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e5b6be27-fa9b-4a94-a518-1c8377b8de07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:30.160515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 757/1477] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:34:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:30.187873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4382e3be-f3d7-47b1-ae41-e599d75f2f41 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:30.190244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4382e3be-f3d7-47b1-ae41-e599d75f2f41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:30.358251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4382e3be-f3d7-47b1-ae41-e599d75f2f41 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:30.426741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:30.492806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51570cb4-7da8-429d-9f58-e54adb50bc5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:30.493487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51570cb4-7da8-429d-9f58-e54adb50bc5f admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:30.493770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51570cb4-7da8-429d-9f58-e54adb50bc5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:30.494050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51570cb4-7da8-429d-9f58-e54adb50bc5f admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:30.494841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51570cb4-7da8-429d-9f58-e54adb50bc5f admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:30.496718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-51570cb4-7da8-429d-9f58-e54adb50bc5f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:34:30.496718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:34:30.496718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:34:30.497893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51570cb4-7da8-429d-9f58-e54adb50bc5f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:30.542466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 721/1478] 10.222.0.72 () {56 vars in 1108 bytes} [Mon Jul 11 14:34:30 2022] GET /identity/v3/projects => generated 2553 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:34:32.763735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2738f5-2eb5-4055-8138-7c6c4b445313 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:32.764297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2738f5-2eb5-4055-8138-7c6c4b445313 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:32.764297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2738f5-2eb5-4055-8138-7c6c4b445313 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:32.765746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 758/1479] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:34:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:32.771721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc5d401-3c71-48c6-860c-a33545e4549b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:32.772263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc5d401-3c71-48c6-860c-a33545e4549b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:32.772263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc5d401-3c71-48c6-860c-a33545e4549b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:32.873648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9dc5d401-3c71-48c6-860c-a33545e4549b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:32.875122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9dc5d401-3c71-48c6-860c-a33545e4549b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:33.034581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 722/1480] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:34:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:33.103398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2738f5-2eb5-4055-8138-7c6c4b445313 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:33.105116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-be2738f5-2eb5-4055-8138-7c6c4b445313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:33.115546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:33.282949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2738f5-2eb5-4055-8138-7c6c4b445313 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:33.401599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:33.453632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bc202d-5d62-4371-9c54-b539d939d9dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:33.454338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bc202d-5d62-4371-9c54-b539d939d9dc service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:33.454702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bc202d-5d62-4371-9c54-b539d939d9dc service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:33.454985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bc202d-5d62-4371-9c54-b539d939d9dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:33.456925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-14bc202d-5d62-4371-9c54-b539d939d9dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:33.570372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bc202d-5d62-4371-9c54-b539d939d9dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:33.572006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bc202d-5d62-4371-9c54-b539d939d9dc service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:33.591628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 759/1481] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:36.000744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db25f6f5-3b82-46b7-b9de-08fa84fe85c8 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:36.001331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db25f6f5-3b82-46b7-b9de-08fa84fe85c8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:36.001723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db25f6f5-3b82-46b7-b9de-08fa84fe85c8 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:36.003572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 723/1482] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:34:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:36.010152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a86d49c-475e-4786-a70e-446a683e5abc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:36.010783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a86d49c-475e-4786-a70e-446a683e5abc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:36.011176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a86d49c-475e-4786-a70e-446a683e5abc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:36.093912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2a86d49c-475e-4786-a70e-446a683e5abc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:36.095164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2a86d49c-475e-4786-a70e-446a683e5abc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:36.146145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:36.220750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 760/1483] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:34:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:34:36.258539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db25f6f5-3b82-46b7-b9de-08fa84fe85c8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:36.260130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-db25f6f5-3b82-46b7-b9de-08fa84fe85c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:36.287083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:36.383785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db25f6f5-3b82-46b7-b9de-08fa84fe85c8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:36.464848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03f6f6f-7965-434e-82d0-d630ec1fa091 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:36.465505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f6f6f-7965-434e-82d0-d630ec1fa091 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:36.465786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f6f6f-7965-434e-82d0-d630ec1fa091 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:36.466061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f6f6f-7965-434e-82d0-d630ec1fa091 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:36.467799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e03f6f6f-7965-434e-82d0-d630ec1fa091 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:36.549022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:36.587986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03f6f6f-7965-434e-82d0-d630ec1fa091 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:36.589470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03f6f6f-7965-434e-82d0-d630ec1fa091 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:36.609381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 724/1484] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:38.792792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0236b962-8ec6-48b4-9c81-a8b66b6e3ae2 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:38.793459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0236b962-8ec6-48b4-9c81-a8b66b6e3ae2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:38.793459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0236b962-8ec6-48b4-9c81-a8b66b6e3ae2 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:38.794806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 761/1485] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:34:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:38.800742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b2157-a0bb-43cf-80bb-e38d7e4e8cfb None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:38.801334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b2157-a0bb-43cf-80bb-e38d7e4e8cfb None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:38.801334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b2157-a0bb-43cf-80bb-e38d7e4e8cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:38.884463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cf6b2157-a0bb-43cf-80bb-e38d7e4e8cfb None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:38.885775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cf6b2157-a0bb-43cf-80bb-e38d7e4e8cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:39.025066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 725/1486] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:34:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:39.071204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0236b962-8ec6-48b4-9c81-a8b66b6e3ae2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:39.073323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0236b962-8ec6-48b4-9c81-a8b66b6e3ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:39.242487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0236b962-8ec6-48b4-9c81-a8b66b6e3ae2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:39.343621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:39.347017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b191f947-2a9a-4ab4-adc4-593175b4a44c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:39.348174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b191f947-2a9a-4ab4-adc4-593175b4a44c service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:39.348942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b191f947-2a9a-4ab4-adc4-593175b4a44c service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:39.350496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b191f947-2a9a-4ab4-adc4-593175b4a44c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:39.353160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b191f947-2a9a-4ab4-adc4-593175b4a44c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:39.512177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b191f947-2a9a-4ab4-adc4-593175b4a44c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:39.513957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b191f947-2a9a-4ab4-adc4-593175b4a44c service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:39.534258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 762/1487] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:41.762362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3106150-b80f-42ce-899a-2ce64ffb16fa None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:41.763225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3106150-b80f-42ce-899a-2ce64ffb16fa None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:41.763225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3106150-b80f-42ce-899a-2ce64ffb16fa None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:41.764988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 726/1488] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:34:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:34:41.770926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf848de2-2ed1-4ee1-a097-e38c9027a64a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:41.771574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf848de2-2ed1-4ee1-a097-e38c9027a64a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:41.771574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf848de2-2ed1-4ee1-a097-e38c9027a64a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:41.848432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-bf848de2-2ed1-4ee1-a097-e38c9027a64a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:41.849842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bf848de2-2ed1-4ee1-a097-e38c9027a64a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:41.917620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:41.968862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 763/1489] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:34:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:42.010106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3106150-b80f-42ce-899a-2ce64ffb16fa None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:42.011769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b3106150-b80f-42ce-899a-2ce64ffb16fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:42.069535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:42.144125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3106150-b80f-42ce-899a-2ce64ffb16fa None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:42.225819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f32da18-c229-454c-a1f0-9854344a60c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:42.226396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f32da18-c229-454c-a1f0-9854344a60c9 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:42.226719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f32da18-c229-454c-a1f0-9854344a60c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:42.227001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f32da18-c229-454c-a1f0-9854344a60c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:42.228727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0f32da18-c229-454c-a1f0-9854344a60c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:42.328123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:42.366798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f32da18-c229-454c-a1f0-9854344a60c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:42.368278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f32da18-c229-454c-a1f0-9854344a60c9 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:42.386869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 727/1490] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:44.709492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3578c-5097-46be-9089-2b73b5c2b863 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:44.710059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3578c-5097-46be-9089-2b73b5c2b863 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:44.710059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3578c-5097-46be-9089-2b73b5c2b863 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:44.711564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 764/1491] 10.222.0.72 () {54 vars in 926 bytes} [Mon Jul 11 14:34:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:34:44.718073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49212635-d16b-4c9c-b4fe-dc7327d1d739 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:44.718643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49212635-d16b-4c9c-b4fe-dc7327d1d739 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:44.719114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49212635-d16b-4c9c-b4fe-dc7327d1d739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:44.812120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-49212635-d16b-4c9c-b4fe-dc7327d1d739 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:34:44.813739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-49212635-d16b-4c9c-b4fe-dc7327d1d739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:44.935082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 728/1492] 10.222.0.72 () {58 vars in 1023 bytes} [Mon Jul 11 14:34:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:34:44.975843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3578c-5097-46be-9089-2b73b5c2b863 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:34:44.977485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e9b3578c-5097-46be-9089-2b73b5c2b863 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:45.103579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3578c-5097-46be-9089-2b73b5c2b863 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:34:45.181003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378f192e-ee6b-49a5-ad7b-4820c7df4062 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:34:45.181664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f192e-ee6b-49a5-ad7b-4820c7df4062 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:34:45.181948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f192e-ee6b-49a5-ad7b-4820c7df4062 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:34:45.182228 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f192e-ee6b-49a5-ad7b-4820c7df4062 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:34:45.183955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-378f192e-ee6b-49a5-ad7b-4820c7df4062 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:34:45.193608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:45.314000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378f192e-ee6b-49a5-ad7b-4820c7df4062 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:34:45.315396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378f192e-ee6b-49a5-ad7b-4820c7df4062 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:34:45.336109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 765/1493] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:34:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:34:54.932148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=82 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:34:55.332997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=50 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:19.006465 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:19.007126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:19.007443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:19.154423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:36:19.156731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:19.348185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 729/1494] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:36:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:36:19.359686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f875df1-4dac-4e1c-9bdf-d8d945e70e25 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:19.360529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f875df1-4dac-4e1c-9bdf-d8d945e70e25 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:19.360806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f875df1-4dac-4e1c-9bdf-d8d945e70e25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:19.483457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-0f875df1-4dac-4e1c-9bdf-d8d945e70e25 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:36:19.484948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0f875df1-4dac-4e1c-9bdf-d8d945e70e25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:19.668994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 766/1495] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:36:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:36:19.676289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:19.678397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:19.699496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7058912-eee6-4648-a42d-dfe0da4ec86c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:19.818379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:19.885047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:19.885723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:19.886011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:19.886292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:19.887085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:36:19.887619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:19.888725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:19.888725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:36:19.888725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:36:19.889906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:19.904151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 730/1496] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:36:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:36:19.917955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f875df1-4dac-4e1c-9bdf-d8d945e70e25 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:19.929322 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f875df1-4dac-4e1c-9bdf-d8d945e70e25 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:20.012562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:20.063950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:20.064898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:20.065350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:20.065792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:20.067538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:20.070524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING py.warnings [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:20.070524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: warnings.warn(msg) Jul 11 14:36:20.070524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: Jul 11 14:36:20.072900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:20.159977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 767/1497] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:36:19 2022] POST /identity/v3/projects => generated 379 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:36:20.164482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:20.166213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:20.209363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09f5db7-ab9c-4bfd-b5f0-d62320dc2279 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:20.343032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:20.344318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:20.344885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:20.345444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:20.346847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:20.348816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:20.348816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:36:20.348816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:36:20.351297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:20.431292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 731/1498] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:36:20 2022] POST /identity/v3/users => generated 290 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:36:20.438465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:20.442040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:20.563002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b469f7-36f5-4e14-a5a4-3f65788b0f69 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:20.676211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:20.690737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:20.691948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:20.692233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:20.692511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:20.693260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:20.694321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING py.warnings [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:20.694321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: warnings.warn(msg) Jul 11 14:36:20.694321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: Jul 11 14:36:20.695403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:20.711728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 768/1499] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:36:20.716515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:20.718564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:20.731035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdf753a-2c28-41b1-ad87-c8ab3d186d9d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:20.760491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:20.862012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24e6fce-91a3-4a76-aff7-2f05988b862b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:20.862643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e6fce-91a3-4a76-aff7-2f05988b862b admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:20.863452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e6fce-91a3-4a76-aff7-2f05988b862b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:20.863734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e6fce-91a3-4a76-aff7-2f05988b862b admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:20.864512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24e6fce-91a3-4a76-aff7-2f05988b862b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:20.867328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-e24e6fce-91a3-4a76-aff7-2f05988b862b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:20.867328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:36:20.867328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:36:20.867328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24e6fce-91a3-4a76-aff7-2f05988b862b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:20.882827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 732/1500] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:36:20.889235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:20.891339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:20.960196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ed3bc9-c053-4992-a6cf-4e856d4d313c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:21.086442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90936d-749f-4943-8285-54472d4aa5be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:21.087370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90936d-749f-4943-8285-54472d4aa5be admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:21.087656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90936d-749f-4943-8285-54472d4aa5be admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:21.087934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90936d-749f-4943-8285-54472d4aa5be admin admin] PATH_INFO: `/v3/projects/8155e94ae0a640d7a79ec6f86590f177/users/a563fea6b8c349b88d8beac0526f5a6d/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:21.113929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90936d-749f-4943-8285-54472d4aa5be admin admin] RBAC: Authorizing `identity:create_grant(project_id=8155e94ae0a640d7a79ec6f86590f177, user_id=a563fea6b8c349b88d8beac0526f5a6d, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:21.114912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING py.warnings [None req-8d90936d-749f-4943-8285-54472d4aa5be admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:21.114912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: warnings.warn(msg) Jul 11 14:36:21.114912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: Jul 11 14:36:21.116892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90936d-749f-4943-8285-54472d4aa5be admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:21.148278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 769/1501] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:36:20 2022] PUT /identity/v3/projects/8155e94ae0a640d7a79ec6f86590f177/users/a563fea6b8c349b88d8beac0526f5a6d/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:36:21.161225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f1ed2c-bb46-4dcf-b103-fed99b8661c8 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:21.161524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f1ed2c-bb46-4dcf-b103-fed99b8661c8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:21.161814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f1ed2c-bb46-4dcf-b103-fed99b8661c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:21.225080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-47f1ed2c-bb46-4dcf-b103-fed99b8661c8 None None] MFA Rules not processed for user `a563fea6b8c349b88d8beac0526f5a6d`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:36:21.226320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-47f1ed2c-bb46-4dcf-b103-fed99b8661c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:21.325939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 733/1502] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:36:21 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:36:21.348540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da6de47-802c-4b99-a881-32c06436d8e8 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:21.349043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da6de47-802c-4b99-a881-32c06436d8e8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:21.349553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da6de47-802c-4b99-a881-32c06436d8e8 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:21.352240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 770/1503] 10.222.0.72 () {54 vars in 963 bytes} [Mon Jul 11 14:36:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:36:21.361275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095940c6-fcfe-46d9-9b1f-c15e476dd40d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:21.361777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095940c6-fcfe-46d9-9b1f-c15e476dd40d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:21.362492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095940c6-fcfe-46d9-9b1f-c15e476dd40d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:21.406572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:21.461036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-095940c6-fcfe-46d9-9b1f-c15e476dd40d None None] MFA Rules not processed for user `4f1b0819d52d481898b354a5ae182746`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:36:21.462250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-095940c6-fcfe-46d9-9b1f-c15e476dd40d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:21.583634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 734/1504] 10.222.0.72 () {58 vars in 1060 bytes} [Mon Jul 11 14:36:21 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:36:21.592475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da6de47-802c-4b99-a881-32c06436d8e8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:21.594804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0da6de47-802c-4b99-a881-32c06436d8e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:21.665522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:21.741895 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da6de47-802c-4b99-a881-32c06436d8e8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:21.829551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:21.830214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:21.830536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:21.830815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:21.832571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:21.955224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:21.956748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:21.960632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:21.981433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 771/1505] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:36:21 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:36:22.035870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8676a8bc-fbae-4ca1-afeb-b45205095610 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:22.036164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8676a8bc-fbae-4ca1-afeb-b45205095610 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:22.036436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8676a8bc-fbae-4ca1-afeb-b45205095610 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:22.037776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 735/1506] 10.222.0.72 () {52 vars in 829 bytes} [Mon Jul 11 14:36:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:36:22.041991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:22.043275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:22.170786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8c1c26-5131-40f6-8113-796ba064bb9d service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:22.275629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:22.276533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:22.276813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:22.277094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] PATH_INFO: `/v3/users/a563fea6b8c349b88d8beac0526f5a6d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:22.291656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a563fea6b8c349b88d8beac0526f5a6d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:22.292704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING py.warnings [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:22.292704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: warnings.warn(msg) Jul 11 14:36:22.292704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: Jul 11 14:36:22.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:22.306570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:22.376985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a563fea6b8c349b88d8beac0526f5a6d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:22.389745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a563fea6b8c349b88d8beac0526f5a6d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:22.390271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a563fea6b8c349b88d8beac0526f5a6d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:22.410697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a563fea6b8c349b88d8beac0526f5a6d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:22.449479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a563fea6b8c349b88d8beac0526f5a6d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:22.461343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 772/1507] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:36:22 2022] DELETE /identity/v3/users/a563fea6b8c349b88d8beac0526f5a6d => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:36:22.466052 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:22.466867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:22.466867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:22.535182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:36:22.536855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:22.772869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:22.788100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 736/1508] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:36:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 323 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:36:22.798906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:22.800166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:22.925811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e1ceb-dacd-4689-9342-824fddbdcd64 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:23.013209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:23.039363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32474953-b423-4d72-b44e-93b82bffaab3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:23.039927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32474953-b423-4d72-b44e-93b82bffaab3 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:23.040218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32474953-b423-4d72-b44e-93b82bffaab3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:23.040501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32474953-b423-4d72-b44e-93b82bffaab3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:23.042214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-32474953-b423-4d72-b44e-93b82bffaab3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:23.189967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32474953-b423-4d72-b44e-93b82bffaab3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:23.193094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32474953-b423-4d72-b44e-93b82bffaab3 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:23.222986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 773/1509] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:36:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:36:23.639705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:36:23.641768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:36:23.670963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b69744-638c-4e01-8089-814bedcb76a1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:36:23.816841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:36:23.817719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:36:23.817995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:36:23.818266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] PATH_INFO: `/v3/projects/8155e94ae0a640d7a79ec6f86590f177` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:36:23.832521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8155e94ae0a640d7a79ec6f86590f177)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:36:23.834337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:36:23.834337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:36:23.834337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:36:23.837852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:36:23.908128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8155e94ae0a640d7a79ec6f86590f177', 'request_id': 'req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:23.915911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8155e94ae0a640d7a79ec6f86590f177', 'request_id': 'req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:23.928700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8155e94ae0a640d7a79ec6f86590f177', 'request_id': 'req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:23.965805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] The token cache is being invalidate because project 8155e94ae0a640d7a79ec6f86590f177 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:36:23.966093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:23.966573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-90906ae1-7e5c-4b38-bd9d-5e1e8eb90e18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:36:23.968081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 737/1510] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:36:23 2022] DELETE /identity/v3/projects/8155e94ae0a640d7a79ec6f86590f177 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:36:33.219726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=62 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:36:33.963684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:38:09.142835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be8747-38a2-4acc-8504-da1dcdcd1fff None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:09.143490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be8747-38a2-4acc-8504-da1dcdcd1fff None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:09.143490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be8747-38a2-4acc-8504-da1dcdcd1fff None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:09.144766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 774/1511] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:38:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:09.149920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9ec3d1-d512-4ac1-9372-7cc53dbe2dd8 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:09.150392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9ec3d1-d512-4ac1-9372-7cc53dbe2dd8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:09.150534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9ec3d1-d512-4ac1-9372-7cc53dbe2dd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:09.237745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1f9ec3d1-d512-4ac1-9372-7cc53dbe2dd8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:09.238918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1f9ec3d1-d512-4ac1-9372-7cc53dbe2dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:09.352055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 738/1512] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:38:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:09.511941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be8747-38a2-4acc-8504-da1dcdcd1fff None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:09.514242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-27be8747-38a2-4acc-8504-da1dcdcd1fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:09.638658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be8747-38a2-4acc-8504-da1dcdcd1fff None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:09.644813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d020bc2-1b3a-4cb7-aa91-0fd27f69fda5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:09.645487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d020bc2-1b3a-4cb7-aa91-0fd27f69fda5 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:09.645755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d020bc2-1b3a-4cb7-aa91-0fd27f69fda5 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:09.646033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d020bc2-1b3a-4cb7-aa91-0fd27f69fda5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:09.654912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d020bc2-1b3a-4cb7-aa91-0fd27f69fda5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:09.656603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d020bc2-1b3a-4cb7-aa91-0fd27f69fda5 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:09.667288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 775/1513] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:10.513782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a86d127-63ee-41db-9373-3d64eee69570 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:10.514373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a86d127-63ee-41db-9373-3d64eee69570 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:10.514373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a86d127-63ee-41db-9373-3d64eee69570 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:10.516326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 739/1514] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:38:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:10.522065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d72291-a7d4-4f67-bf45-c4a86397ed7c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:10.522703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d72291-a7d4-4f67-bf45-c4a86397ed7c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:10.522703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d72291-a7d4-4f67-bf45-c4a86397ed7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:10.544168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c1d72291-a7d4-4f67-bf45-c4a86397ed7c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:10.545520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c1d72291-a7d4-4f67-bf45-c4a86397ed7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:10.553796 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 776/1515] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:38:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:38:10.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a86d127-63ee-41db-9373-3d64eee69570 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:10.734619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a86d127-63ee-41db-9373-3d64eee69570 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:10.740999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f28089-b47f-49e7-bd0c-a502d944dd94 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:10.741665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f28089-b47f-49e7-bd0c-a502d944dd94 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:10.741962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f28089-b47f-49e7-bd0c-a502d944dd94 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:10.742237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f28089-b47f-49e7-bd0c-a502d944dd94 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:10.752259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f28089-b47f-49e7-bd0c-a502d944dd94 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:10.753883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f28089-b47f-49e7-bd0c-a502d944dd94 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:10.765134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 740/1516] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:11.003351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac199d1-f4ee-4c16-8911-04daa4b51f59 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:11.003641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac199d1-f4ee-4c16-8911-04daa4b51f59 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:11.003924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac199d1-f4ee-4c16-8911-04daa4b51f59 None None] PATH_INFO: `/v3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:11.005289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 777/1517] 10.222.0.36 () {54 vars in 925 bytes} [Mon Jul 11 14:38:11 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:38:11.011766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c8f84f-3bc2-4da0-8c78-46d5f812c729 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:11.012085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c8f84f-3bc2-4da0-8c78-46d5f812c729 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:11.012410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c8f84f-3bc2-4da0-8c78-46d5f812c729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:11.086667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-18c8f84f-3bc2-4da0-8c78-46d5f812c729 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:11.087880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-18c8f84f-3bc2-4da0-8c78-46d5f812c729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:11.110473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:38:11.177635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 741/1518] 10.222.0.36 () {58 vars in 1015 bytes} [Mon Jul 11 14:38:11 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:15.004317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf139d-3e29-45db-96c1-e2022e902d34 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:15.004905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf139d-3e29-45db-96c1-e2022e902d34 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:15.004905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf139d-3e29-45db-96c1-e2022e902d34 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:15.006303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 778/1519] 10.222.0.72 () {54 vars in 960 bytes} [Mon Jul 11 14:38:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:15.012278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df2105-5a25-44ec-8abf-effd769ecc77 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:15.012803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df2105-5a25-44ec-8abf-effd769ecc77 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:15.012922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df2105-5a25-44ec-8abf-effd769ecc77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:15.032866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a3df2105-5a25-44ec-8abf-effd769ecc77 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:15.034081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a3df2105-5a25-44ec-8abf-effd769ecc77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:15.042673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 742/1520] 10.222.0.72 () {58 vars in 1057 bytes} [Mon Jul 11 14:38:15 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:15.050744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf139d-3e29-45db-96c1-e2022e902d34 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:15.064590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf139d-3e29-45db-96c1-e2022e902d34 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:15.069582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f12be36-53ba-4320-9e4b-62de782cd3e0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:15.070226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12be36-53ba-4320-9e4b-62de782cd3e0 service placement] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:15.070559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12be36-53ba-4320-9e4b-62de782cd3e0 service placement] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:15.070840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12be36-53ba-4320-9e4b-62de782cd3e0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:15.080489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f12be36-53ba-4320-9e4b-62de782cd3e0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:15.082081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f12be36-53ba-4320-9e4b-62de782cd3e0 service placement] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:15.092117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 779/1521] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:22.038967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63308d3f-c150-4740-bb85-ff9c1dd3876d None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:22.039608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63308d3f-c150-4740-bb85-ff9c1dd3876d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:22.039608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63308d3f-c150-4740-bb85-ff9c1dd3876d None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:22.041061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 743/1522] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:38:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:22.047593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5205bd7-3815-43f7-9bf7-812447d94c5a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:22.048183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5205bd7-3815-43f7-9bf7-812447d94c5a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:22.048183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5205bd7-3815-43f7-9bf7-812447d94c5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:22.068217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c5205bd7-3815-43f7-9bf7-812447d94c5a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:22.069442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c5205bd7-3815-43f7-9bf7-812447d94c5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:22.077486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 780/1523] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:38:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:22.233262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63308d3f-c150-4740-bb85-ff9c1dd3876d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:22.248712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63308d3f-c150-4740-bb85-ff9c1dd3876d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:22.254652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853d5da4-72d4-4b39-9bc4-9c6c325b4abe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:22.255370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853d5da4-72d4-4b39-9bc4-9c6c325b4abe service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:22.255654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853d5da4-72d4-4b39-9bc4-9c6c325b4abe service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:22.255939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853d5da4-72d4-4b39-9bc4-9c6c325b4abe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:22.265373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853d5da4-72d4-4b39-9bc4-9c6c325b4abe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:22.267025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853d5da4-72d4-4b39-9bc4-9c6c325b4abe service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:22.277630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 744/1524] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:23.115730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e405a-902f-4b7e-a9ad-168f2f4ff660 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:23.116351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e405a-902f-4b7e-a9ad-168f2f4ff660 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:23.116351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e405a-902f-4b7e-a9ad-168f2f4ff660 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:23.118520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 781/1525] 127.0.0.1 () {54 vars in 901 bytes} [Mon Jul 11 14:38:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:23.123964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca2971-ad9d-4ee6-821a-d23cc845f8ca None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:23.124467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca2971-ad9d-4ee6-821a-d23cc845f8ca None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:23.124564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca2971-ad9d-4ee6-821a-d23cc845f8ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:23.144153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9bca2971-ad9d-4ee6-821a-d23cc845f8ca None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:23.145343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9bca2971-ad9d-4ee6-821a-d23cc845f8ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:23.153476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 745/1526] 127.0.0.1 () {58 vars in 998 bytes} [Mon Jul 11 14:38:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:23.314983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7e405a-902f-4b7e-a9ad-168f2f4ff660 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:23.330572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7e405a-902f-4b7e-a9ad-168f2f4ff660 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:23.336143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0da6906-02ac-41d7-8369-2e52a922b74f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:23.336790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da6906-02ac-41d7-8369-2e52a922b74f service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:23.337077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da6906-02ac-41d7-8369-2e52a922b74f service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:23.337358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da6906-02ac-41d7-8369-2e52a922b74f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:23.346591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0da6906-02ac-41d7-8369-2e52a922b74f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:23.348232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0da6906-02ac-41d7-8369-2e52a922b74f service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:23.357793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 782/1527] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:27.497950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2792c189-b608-4045-83b7-d122ce8e1c61 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:27.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2792c189-b608-4045-83b7-d122ce8e1c61 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:27.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2792c189-b608-4045-83b7-d122ce8e1c61 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:27.500554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 746/1528] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:38:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:27.506303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3dbfe9-ab47-45b6-b334-8e48ed8b5684 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:27.506833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3dbfe9-ab47-45b6-b334-8e48ed8b5684 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:27.506939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3dbfe9-ab47-45b6-b334-8e48ed8b5684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:27.525455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ce3dbfe9-ab47-45b6-b334-8e48ed8b5684 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:27.526707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ce3dbfe9-ab47-45b6-b334-8e48ed8b5684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:27.534778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 783/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:38:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:27.614150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2792c189-b608-4045-83b7-d122ce8e1c61 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:27.616150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2792c189-b608-4045-83b7-d122ce8e1c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:27.699282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2792c189-b608-4045-83b7-d122ce8e1c61 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:27.703008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e837eff1-4a9b-4233-ac3b-000beeab490c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:27.703661 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837eff1-4a9b-4233-ac3b-000beeab490c service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:27.703973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837eff1-4a9b-4233-ac3b-000beeab490c service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:27.704487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837eff1-4a9b-4233-ac3b-000beeab490c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:27.714448 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e837eff1-4a9b-4233-ac3b-000beeab490c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:27.715933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e837eff1-4a9b-4233-ac3b-000beeab490c service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:27.725498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 747/1530] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:29.810187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a6ce2-b143-49af-9788-d5a8921d0dff None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:29.810794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a6ce2-b143-49af-9788-d5a8921d0dff None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:29.810794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a6ce2-b143-49af-9788-d5a8921d0dff None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:29.812175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 784/1531] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:38:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:29.817450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2e388c-4f46-4eb3-b58f-e12c3de60458 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:29.817871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2e388c-4f46-4eb3-b58f-e12c3de60458 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:29.818060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2e388c-4f46-4eb3-b58f-e12c3de60458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:29.836462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-4c2e388c-4f46-4eb3-b58f-e12c3de60458 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:29.837678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4c2e388c-4f46-4eb3-b58f-e12c3de60458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:29.846308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 748/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:38:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:29.882510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a6ce2-b143-49af-9788-d5a8921d0dff None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:29.894516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a6ce2-b143-49af-9788-d5a8921d0dff None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:29.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f417d8-4d69-4778-ae7b-58b4829e52ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:29.898767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f417d8-4d69-4778-ae7b-58b4829e52ae service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:29.899048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f417d8-4d69-4778-ae7b-58b4829e52ae service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:29.899326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f417d8-4d69-4778-ae7b-58b4829e52ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:29.909943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f417d8-4d69-4778-ae7b-58b4829e52ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:29.911472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f417d8-4d69-4778-ae7b-58b4829e52ae service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:29.921475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 785/1533] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:33.527423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7475d0-6400-4cd8-863d-26624c259e2c None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:33.528037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7475d0-6400-4cd8-863d-26624c259e2c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:33.528037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7475d0-6400-4cd8-863d-26624c259e2c None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:33.529429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 749/1534] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:38:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:33.535272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06de60c0-2df1-4ca0-b59a-97b76ddef8b1 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:33.535703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06de60c0-2df1-4ca0-b59a-97b76ddef8b1 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:33.535931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06de60c0-2df1-4ca0-b59a-97b76ddef8b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:33.542512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=81 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:38:33.556273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-06de60c0-2df1-4ca0-b59a-97b76ddef8b1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:33.557632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-06de60c0-2df1-4ca0-b59a-97b76ddef8b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:33.566723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 786/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:38:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:33.897370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7475d0-6400-4cd8-863d-26624c259e2c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:33.899102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ec7475d0-6400-4cd8-863d-26624c259e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:33.968776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:38:33.982845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7475d0-6400-4cd8-863d-26624c259e2c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:33.987092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef093275-00d9-404c-8bc4-3635ede81d21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:33.987758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef093275-00d9-404c-8bc4-3635ede81d21 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:33.988045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef093275-00d9-404c-8bc4-3635ede81d21 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:33.988329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef093275-00d9-404c-8bc4-3635ede81d21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:33.997967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef093275-00d9-404c-8bc4-3635ede81d21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:34.000142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef093275-00d9-404c-8bc4-3635ede81d21 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:34.010976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 750/1536] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:36.121165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2373d73-bc1a-44f8-98dd-a52644ffc512 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:36.121808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2373d73-bc1a-44f8-98dd-a52644ffc512 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:36.121808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2373d73-bc1a-44f8-98dd-a52644ffc512 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:36.123164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 787/1537] 127.0.0.1 () {54 vars in 916 bytes} [Mon Jul 11 14:38:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:38:36.128774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c98a6-8f1e-4898-9407-3f7ac866752d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:36.129269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c98a6-8f1e-4898-9407-3f7ac866752d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:36.129366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c98a6-8f1e-4898-9407-3f7ac866752d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:36.148118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fd4c98a6-8f1e-4898-9407-3f7ac866752d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:38:36.149343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fd4c98a6-8f1e-4898-9407-3f7ac866752d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:36.158331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 751/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Mon Jul 11 14:38:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:38:36.468807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2373d73-bc1a-44f8-98dd-a52644ffc512 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:38:36.470640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b2373d73-bc1a-44f8-98dd-a52644ffc512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:38:36.484919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2373d73-bc1a-44f8-98dd-a52644ffc512 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:38:36.488522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fe3975-df8a-4b49-a7a0-faf8c50d1efe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:38:36.489177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe3975-df8a-4b49-a7a0-faf8c50d1efe service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:38:36.489464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe3975-df8a-4b49-a7a0-faf8c50d1efe service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:38:36.489744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fe3975-df8a-4b49-a7a0-faf8c50d1efe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:38:36.500889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fe3975-df8a-4b49-a7a0-faf8c50d1efe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:38:36.502353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fe3975-df8a-4b49-a7a0-faf8c50d1efe service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:38:36.512610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 788/1539] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:38:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:38:46.143087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=11 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:38:46.508254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=8 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:06.908826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c24e36-d64e-46fb-96b7-d2fac500d21d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:06.920724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c24e36-d64e-46fb-96b7-d2fac500d21d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:06.920724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c24e36-d64e-46fb-96b7-d2fac500d21d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:06.933301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-92c24e36-d64e-46fb-96b7-d2fac500d21d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:06.934992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-92c24e36-d64e-46fb-96b7-d2fac500d21d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:06.951299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 752/1540] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:06.961143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314f2b63-5a4d-4c52-9f73-b3cc2dd7039a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:06.961729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314f2b63-5a4d-4c52-9f73-b3cc2dd7039a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:06.961837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314f2b63-5a4d-4c52-9f73-b3cc2dd7039a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:06.991756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-314f2b63-5a4d-4c52-9f73-b3cc2dd7039a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:06.993079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-314f2b63-5a4d-4c52-9f73-b3cc2dd7039a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:07.002555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 789/1541] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:07.007584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c24e36-d64e-46fb-96b7-d2fac500d21d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.018352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c24e36-d64e-46fb-96b7-d2fac500d21d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.031330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.032027 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.032315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.032594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.033498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:07.033814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.035443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.052869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 753/1542] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:07.062252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314f2b63-5a4d-4c52-9f73-b3cc2dd7039a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.072441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314f2b63-5a4d-4c52-9f73-b3cc2dd7039a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.083119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.083955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.084250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.084532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.085243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.087203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.139798 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 790/1543] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/projects => generated 371 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:07.144736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.159809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dcc2b5-d321-4615-b793-e4b55662de69 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.258263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.259008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.259297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.259576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.260285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.263453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.294192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 754/1544] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/users => generated 286 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:07.310322 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.325990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad854e-8837-454d-b823-527c7cfe88bc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.337907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.338635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.338930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.339219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.339975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.341723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.366311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 791/1545] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:07 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:07.370880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.385583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee61ac0-b43d-45b5-ab07-63368ffc1611 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.396315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75170a-ba7b-4665-b892-6fcb3de24e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.397132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75170a-ba7b-4665-b892-6fcb3de24e0a admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.397439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75170a-ba7b-4665-b892-6fcb3de24e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.397720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75170a-ba7b-4665-b892-6fcb3de24e0a admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.398859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e75170a-ba7b-4665-b892-6fcb3de24e0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.401770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e75170a-ba7b-4665-b892-6fcb3de24e0a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.420737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 755/1546] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:07 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:07.425581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.442194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdcdd5c-d5d4-4969-b000-2672c978dbfc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.452799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.453709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.453996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.454277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] PATH_INFO: `/v3/projects/7eba23022dd5473a963ee8125858ccfa/users/f11d3a5af8f1442ab935343809610f9f/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.479106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eba23022dd5473a963ee8125858ccfa, user_id=f11d3a5af8f1442ab935343809610f9f, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.482165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.505161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 792/1547] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:07 2022] PUT /identity/v3/projects/7eba23022dd5473a963ee8125858ccfa/users/f11d3a5af8f1442ab935343809610f9f/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:07.528465 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d999817-b0a9-4ed3-a8c2-4515bdaf358d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.528671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d999817-b0a9-4ed3-a8c2-4515bdaf358d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.529068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d999817-b0a9-4ed3-a8c2-4515bdaf358d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.568738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1d999817-b0a9-4ed3-a8c2-4515bdaf358d None None] MFA Rules not processed for user `f11d3a5af8f1442ab935343809610f9f`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:07.570078 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1d999817-b0a9-4ed3-a8c2-4515bdaf358d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:07.653173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 756/1548] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:07.664667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.725409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2817c56-4831-46e6-9d89-a420bdbe1f25 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.765821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2817c56-4831-46e6-9d89-a420bdbe1f25 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.765821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2817c56-4831-46e6-9d89-a420bdbe1f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.765821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f2817c56-4831-46e6-9d89-a420bdbe1f25 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:07.765821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f2817c56-4831-46e6-9d89-a420bdbe1f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:07.777382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 757/1549] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:07.793222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5196da7f-d64f-4a5e-a537-5f7f9344d17d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.793462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6ca5b-d661-4953-ba3b-ef755101a1e4 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.793462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6ca5b-d661-4953-ba3b-ef755101a1e4 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.793462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6ca5b-d661-4953-ba3b-ef755101a1e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.795430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.796145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.796418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.797310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.797626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.820386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1fb6ca5b-d661-4953-ba3b-ef755101a1e4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:07.821804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1fb6ca5b-d661-4953-ba3b-ef755101a1e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:07.838043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 758/1550] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:40:07.841985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1900d34a-a552-46d0-ae45-0e69a1b257fd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.842294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1900d34a-a552-46d0-ae45-0e69a1b257fd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.842623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1900d34a-a552-46d0-ae45-0e69a1b257fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.855527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 793/1551] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/projects => generated 369 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:07.857028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.872421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1900d34a-a552-46d0-ae45-0e69a1b257fd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:07.874293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1900d34a-a552-46d0-ae45-0e69a1b257fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:07.940965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:07.949024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339515d3-893e-4f5b-a9c9-87252ed39797 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:07.959018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:07.959708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.959999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.960283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:07.961345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:07.961928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:07.964369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING py.warnings [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:40:07.964369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: warnings.warn(msg) Jul 11 14:40:07.964369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: Jul 11 14:40:07.965375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573a9ef9-d6cd-4b98-a3f8-4044494e81e0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:07.994017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 759/1552] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:07.994696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1900d34a-a552-46d0-ae45-0e69a1b257fd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:07.995469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 794/1553] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:07.996858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed04f47f-73bc-486a-b125-77e908d72317 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:07.997158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed04f47f-73bc-486a-b125-77e908d72317 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:07.997445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed04f47f-73bc-486a-b125-77e908d72317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.013988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1900d34a-a552-46d0-ae45-0e69a1b257fd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.021777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:08.023200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611d0acc-5746-4972-8038-e679cc4f2e8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.024140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d0acc-5746-4972-8038-e679cc4f2e8b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.024503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d0acc-5746-4972-8038-e679cc4f2e8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.024847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d0acc-5746-4972-8038-e679cc4f2e8b admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.026803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611d0acc-5746-4972-8038-e679cc4f2e8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.030024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ed04f47f-73bc-486a-b125-77e908d72317 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.030179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611d0acc-5746-4972-8038-e679cc4f2e8b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.031906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ed04f47f-73bc-486a-b125-77e908d72317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.040568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 795/1554] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.045728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2728e-4f0e-4c7a-be82-3fed63d6feb3 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.045970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2728e-4f0e-4c7a-be82-3fed63d6feb3 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.046303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2728e-4f0e-4c7a-be82-3fed63d6feb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.056163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 760/1555] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:07 2022] POST /identity/v3/users => generated 291 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.056163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3a385-8bcf-4874-849a-ea8f67088297 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.056163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3a385-8bcf-4874-849a-ea8f67088297 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.056163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3a385-8bcf-4874-849a-ea8f67088297 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.073676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d0d2728e-4f0e-4c7a-be82-3fed63d6feb3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.075078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d0d2728e-4f0e-4c7a-be82-3fed63d6feb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.075657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9af3a385-8bcf-4874-849a-ea8f67088297 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.076833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9af3a385-8bcf-4874-849a-ea8f67088297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.090466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 796/1556] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.090466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43429599-337d-4869-815e-3e788f888cfc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.090466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43429599-337d-4869-815e-3e788f888cfc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.090466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43429599-337d-4869-815e-3e788f888cfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.093418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 761/1557] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.095471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af3a385-8bcf-4874-849a-ea8f67088297 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.106930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-43429599-337d-4869-815e-3e788f888cfc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.108136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-43429599-337d-4869-815e-3e788f888cfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.109766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af3a385-8bcf-4874-849a-ea8f67088297 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.116084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877402b-0ec0-4880-a428-7875f863e27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.117039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877402b-0ec0-4880-a428-7875f863e27d admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.117176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 797/1558] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.117264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877402b-0ec0-4880-a428-7875f863e27d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.117474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877402b-0ec0-4880-a428-7875f863e27d admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.118203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3bc087-5b9a-487d-bd19-8a359c4dd769 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.118490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3bc087-5b9a-487d-bd19-8a359c4dd769 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.118611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877402b-0ec0-4880-a428-7875f863e27d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.118772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3bc087-5b9a-487d-bd19-8a359c4dd769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.119695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-4877402b-0ec0-4880-a428-7875f863e27d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:40:08.119695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:40:08.119695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:40:08.121200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877402b-0ec0-4880-a428-7875f863e27d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.142038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-eb3bc087-5b9a-487d-bd19-8a359c4dd769 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.143467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-eb3bc087-5b9a-487d-bd19-8a359c4dd769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.151841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 762/1559] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/projects => generated 383 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.152136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 798/1560] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.152951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3bc087-5b9a-487d-bd19-8a359c4dd769 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.160159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acf28ae-ce6c-4164-86fb-2f2a6df8db77 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.160356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acf28ae-ce6c-4164-86fb-2f2a6df8db77 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.160633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acf28ae-ce6c-4164-86fb-2f2a6df8db77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.164850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3bc087-5b9a-487d-bd19-8a359c4dd769 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.179598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1acf28ae-ce6c-4164-86fb-2f2a6df8db77 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.180797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1acf28ae-ce6c-4164-86fb-2f2a6df8db77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.226229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4f4732-d836-444a-949b-fdbe81829dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.226972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f4732-d836-444a-949b-fdbe81829dd2 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.227260 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f4732-d836-444a-949b-fdbe81829dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.227544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f4732-d836-444a-949b-fdbe81829dd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.228276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4f4732-d836-444a-949b-fdbe81829dd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.229897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4f4732-d836-444a-949b-fdbe81829dd2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.244131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 799/1561] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:08.244131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc9b9d-f677-4b74-b388-7ae456bdc022 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.244430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 763/1562] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.244524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc9b9d-f677-4b74-b388-7ae456bdc022 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.244524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc9b9d-f677-4b74-b388-7ae456bdc022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.244725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78752cc6-0cc4-487c-be13-579e5a2761f8 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.244725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78752cc6-0cc4-487c-be13-579e5a2761f8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.244725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78752cc6-0cc4-487c-be13-579e5a2761f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.262037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f7fc9b9d-f677-4b74-b388-7ae456bdc022 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.262266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-78752cc6-0cc4-487c-be13-579e5a2761f8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.263276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-78752cc6-0cc4-487c-be13-579e5a2761f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.263545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f7fc9b9d-f677-4b74-b388-7ae456bdc022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.271338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 764/1563] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.272215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78752cc6-0cc4-487c-be13-579e5a2761f8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.279058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 800/1564] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.279058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9f060-e100-4c97-9784-ed62206866ab None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.279058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9f060-e100-4c97-9784-ed62206866ab None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.279058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9f060-e100-4c97-9784-ed62206866ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.288269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78752cc6-0cc4-487c-be13-579e5a2761f8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.292895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e70eaab-6b9f-45dc-a0f1-d5c1fe9869d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.294459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70eaab-6b9f-45dc-a0f1-d5c1fe9869d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.294738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70eaab-6b9f-45dc-a0f1-d5c1fe9869d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.295013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70eaab-6b9f-45dc-a0f1-d5c1fe9869d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.295790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e70eaab-6b9f-45dc-a0f1-d5c1fe9869d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:08.296047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e70eaab-6b9f-45dc-a0f1-d5c1fe9869d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.296758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-16b9f060-e100-4c97-9784-ed62206866ab None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.297638 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e70eaab-6b9f-45dc-a0f1-d5c1fe9869d4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.297966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-16b9f060-e100-4c97-9784-ed62206866ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.306204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 801/1565] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.306826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b9f060-e100-4c97-9784-ed62206866ab None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.310601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 765/1566] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:08.312090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959dd2db-43f5-4053-8147-9387835903ff None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.312383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959dd2db-43f5-4053-8147-9387835903ff None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.312658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959dd2db-43f5-4053-8147-9387835903ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.321165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b9f060-e100-4c97-9784-ed62206866ab None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.325786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.326533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.326826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.327104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.327857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.328891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING py.warnings [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:40:08.328891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: warnings.warn(msg) Jul 11 14:40:08.328891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: Jul 11 14:40:08.330369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.330720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-959dd2db-43f5-4053-8147-9387835903ff None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.331924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-959dd2db-43f5-4053-8147-9387835903ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.340810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 766/1567] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.341600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959dd2db-43f5-4053-8147-9387835903ff None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.351690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959dd2db-43f5-4053-8147-9387835903ff None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.355481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 802/1568] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/users => generated 292 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.356683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.356839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.357731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.358249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.358377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] PATH_INFO: `/v3/projects/7e9dbf4ac1d549828059d61faabd4ccc/users/ac7147b8e31841f8915cfaf4fedb07e3/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.368360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7547a48a-a7f9-4113-ada0-92dac579795b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.373255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.373873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e9dbf4ac1d549828059d61faabd4ccc, user_id=ac7147b8e31841f8915cfaf4fedb07e3, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.373988 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.374183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.374479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.374950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:40:08.374950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:40:08.374950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:40:08.375221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:08.375513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.376870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.377104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.391277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 803/1569] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:08.395397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.395565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 767/1570] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:08 2022] PUT /identity/v3/projects/7e9dbf4ac1d549828059d61faabd4ccc/users/ac7147b8e31841f8915cfaf4fedb07e3/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:08.396424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.408744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82636e84-68f8-44aa-99f0-d1c2912babe5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.439606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:08.461419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec6cf9-9707-4a71-9d5c-db2892e3e812 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.466620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395359b0-f58e-424c-a751-d559243c7a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.466777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.467287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395359b0-f58e-424c-a751-d559243c7a00 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.467412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.467573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395359b0-f58e-424c-a751-d559243c7a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.467723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.467845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395359b0-f58e-424c-a751-d559243c7a00 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.467937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.468590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395359b0-f58e-424c-a751-d559243c7a00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:08.468703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:08.468900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395359b0-f58e-424c-a751-d559243c7a00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.469015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.470561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.470861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395359b0-f58e-424c-a751-d559243c7a00 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.484396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 768/1571] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:08.485490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 804/1572] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:08.485837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4b7fdb-d2e8-4540-bf9f-552753c533b1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.486192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4b7fdb-d2e8-4540-bf9f-552753c533b1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.486495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4b7fdb-d2e8-4540-bf9f-552753c533b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.487238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.497974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3610b-e8ba-44f2-b8ec-885d214b3d67 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.503582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.504241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.504522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.504799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.505936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.507882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.510825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-0f4b7fdb-d2e8-4540-bf9f-552753c533b1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:08.512071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0f4b7fdb-d2e8-4540-bf9f-552753c533b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:08.520264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 769/1573] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:08.521081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4b7fdb-d2e8-4540-bf9f-552753c533b1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.532879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4b7fdb-d2e8-4540-bf9f-552753c533b1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.536860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 805/1574] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/projects => generated 393 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.537696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.539479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.540393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.540680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.540963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.541872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:08.542166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.544191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:08.545123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.558695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 770/1575] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:08.559526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.571201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeb444c-af08-44ca-8442-bb8a4c50eb36 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.603736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27901cc4-0d4e-4347-a1cd-272a2a54c57a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.608929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.609639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.609930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.610215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.610981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.612599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.622932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 806/1576] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:08.626682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.628108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.628777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.629062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.629341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.630037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.632002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.636332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f923ad0b-f6a8-48ad-b234-7898c1474b79 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.641514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.642168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.642486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.642782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.643523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.645125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.655664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 807/1577] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:08.659661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.663509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 771/1578] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/projects => generated 389 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.664194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.670628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52694f4c-e7d8-4850-aad4-82b4926934fa admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.729466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.730143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.730490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.731568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.732307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.733850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f498e889-1433-479e-8950-d3ed645c0ecc admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.734224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.738660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.739333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.739624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.739900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.740602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.742505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.760560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 808/1579] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/projects => generated 377 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.761490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.778908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 772/1580] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/projects => generated 383 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.778908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.803099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:08.829951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f950c53-725e-44eb-9ba8-16c3e6a83bfc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.841712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05023d80-9615-4148-84a2-afb58c2a70b4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.846373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.847114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.847379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.847712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.848433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.849911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:08.850529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.872342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 773/1581] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/users => generated 297 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.876299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.886033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92201db-61d1-41e8-946b-a97fc89776b4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.891380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.892043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.892335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.892637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.893325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.893440 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.894089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.894379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.894691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.895428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:08.895648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.895766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.897374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.909999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 809/1582] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:08.918517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.926099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 774/1583] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:08 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:08.926099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:08.928749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03e8661-4608-4132-9976-886438d6ba91 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.989858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772400d3-972a-4d36-9898-93b99e037f30 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:08.990496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.990885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.990885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.991157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.991891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:08.993506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:08.994848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:08.995746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:08.996034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:08.996874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] PATH_INFO: `/v3/projects/7e9dbf4ac1d549828059d61faabd4ccc/users/ac7147b8e31841f8915cfaf4fedb07e3/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:08.998950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e9dbf4ac1d549828059d61faabd4ccc, user_id=ac7147b8e31841f8915cfaf4fedb07e3, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.001613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.004232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 810/1584] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:08 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.005058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.014644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 775/1585] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:08 2022] PUT /identity/v3/projects/7e9dbf4ac1d549828059d61faabd4ccc/users/ac7147b8e31841f8915cfaf4fedb07e3/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:09.021297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.021432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dfcf12-f923-4fec-9f3f-268675ae712f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.085535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.086140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.086448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.086740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.087440 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.089402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.090772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2605066-ec08-4200-8b93-0ca2e0318421 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.095382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.096053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.096338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.096609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.097295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.099233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.110273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 811/1586] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/users => generated 295 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:09.111238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.121915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4f217f-1888-4442-a501-bbfb139542f6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.126671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 776/1587] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/users => generated 289 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:09.126671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.130043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.130764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.131054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.131339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.132044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.134041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.134733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369ff261-62f2-45b3-9c94-b96df1b90519 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.139901 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.140566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.140946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.141149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.141878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.143507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.154966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 777/1588] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.155870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.160429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 812/1589] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/users => generated 292 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:09.161201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.165198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:09.169341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2b7b01-31c6-4e27-b02a-9b086790a4f2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.173369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034eedf6-bdbb-4a59-a629-cf78ba40f2cf admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.175570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.176230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.176517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.176795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.177526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.179338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c60223f-e3b8-447f-97f1-c5643c60a95a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.179554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.180175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c60223f-e3b8-447f-97f1-c5643c60a95a admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.180568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c60223f-e3b8-447f-97f1-c5643c60a95a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.180891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c60223f-e3b8-447f-97f1-c5643c60a95a admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.181717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c60223f-e3b8-447f-97f1-c5643c60a95a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.184670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c60223f-e3b8-447f-97f1-c5643c60a95a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.207878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 778/1590] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:09.208480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 813/1591] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/users => generated 287 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:09.208711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.209760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a098a7ac-18cc-44e8-8b6a-2b79b43394ff None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.210050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a098a7ac-18cc-44e8-8b6a-2b79b43394ff None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.210335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a098a7ac-18cc-44e8-8b6a-2b79b43394ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.222738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:09.239453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a098a7ac-18cc-44e8-8b6a-2b79b43394ff None None] MFA Rules not processed for user `ac7147b8e31841f8915cfaf4fedb07e3`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:09.240701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a098a7ac-18cc-44e8-8b6a-2b79b43394ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:09.287495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693663d5-e92b-4074-a8a1-0172621f41a0 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.293156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.294746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.295032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.295312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] PATH_INFO: `/v3/projects/bf9d11f0b33a402f8584b4a158b3b41b/users/072255e7ae5147cb9a1540fd12f242da/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.310877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf9d11f0b33a402f8584b4a158b3b41b, user_id=072255e7ae5147cb9a1540fd12f242da, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.313549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.314171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 814/1592] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:09.314872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a098a7ac-18cc-44e8-8b6a-2b79b43394ff None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.327273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a098a7ac-18cc-44e8-8b6a-2b79b43394ff None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.329173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 779/1593] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:09 2022] PUT /identity/v3/projects/bf9d11f0b33a402f8584b4a158b3b41b/users/072255e7ae5147cb9a1540fd12f242da/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:09.330338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.386876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.387978 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.388267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.388548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.389304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.390958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.404717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 815/1594] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.404902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd18e18-4de5-44a2-b5c7-e07840c526a8 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.405032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.409522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.410208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.410519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.410802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.411546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.412530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8250499-28af-487c-8182-1dfbcdc43f49 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.413160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.417461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.418123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.418594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.418880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.419609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.421236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.423792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 780/1595] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.427866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.431693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 816/1596] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.432502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.441441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd69b253-473a-4a49-92d6-854ccea39dfa admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.442237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51605f0a-f1dd-43a2-aeef-e979b70621c9 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.446054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.446779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.446908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.447489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.447799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.447921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.448095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.448195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.448760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.448869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.451029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.451234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.461840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 781/1597] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.462750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.473697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3da94d8-e691-477d-90a8-be6ac9b1a80d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.474662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 817/1598] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/users => generated 288 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:09.475495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.481391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c81faf-43a8-4134-aedb-9a89f8eb86c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.482040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c81faf-43a8-4134-aedb-9a89f8eb86c9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.482332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c81faf-43a8-4134-aedb-9a89f8eb86c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.482644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c81faf-43a8-4134-aedb-9a89f8eb86c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.483376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c81faf-43a8-4134-aedb-9a89f8eb86c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.484989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c81faf-43a8-4134-aedb-9a89f8eb86c9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.495994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 782/1599] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.497376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663bbcd0-9878-4b7d-86d3-a6d2c8c0f4f3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.497662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663bbcd0-9878-4b7d-86d3-a6d2c8c0f4f3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.497942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663bbcd0-9878-4b7d-86d3-a6d2c8c0f4f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.524749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-663bbcd0-9878-4b7d-86d3-a6d2c8c0f4f3 None None] MFA Rules not processed for user `072255e7ae5147cb9a1540fd12f242da`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:09.525969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-663bbcd0-9878-4b7d-86d3-a6d2c8c0f4f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:09.534319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c748e8-503b-4dcf-8d6d-8049da9fc540 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.538107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.538794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.539078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.539426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.552303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:09.556952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:09.597125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 783/1600] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:09.598347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663bbcd0-9878-4b7d-86d3-a6d2c8c0f4f3 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.608757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.610290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.611012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663bbcd0-9878-4b7d-86d3-a6d2c8c0f4f3 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.615965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.616784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.616910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.617188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.617911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.619554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.621565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 818/1601] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:09.623512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.634786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 784/1602] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.635867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.641004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21771ce8-aed2-454d-967e-6509f8f9aba4 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.647029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.647766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.648085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.648299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcafd57a-bc71-42e9-82d9-fc1ae1560399 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.648530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.649136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.651084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.653709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.654815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.655102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.655382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] PATH_INFO: `/v3/projects/98966d44ea88477b94215761cdc92ea7/users/fa478fb4b41e49c5b1f9291f925cca23/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.663738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 819/1603] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.664623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.677020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9170f509-51d0-4338-bb78-7581e557a161 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.679495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] RBAC: Authorizing `identity:create_grant(project_id=98966d44ea88477b94215761cdc92ea7, user_id=fa478fb4b41e49c5b1f9291f925cca23, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.682189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.682472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.683261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.683619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.684056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.685048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.687477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.697937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 785/1604] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:09 2022] PUT /identity/v3/projects/98966d44ea88477b94215761cdc92ea7/users/fa478fb4b41e49c5b1f9291f925cca23/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:09.698624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.706998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 820/1605] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.706998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.719364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4800af61-36e9-416b-9362-c82e3f351371 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.774563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65bcdc5-dfb1-43ad-9ca6-01caaf1d178b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.778573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83774ae2-8ca3-4c84-8e50-764be14e4ad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.779252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83774ae2-8ca3-4c84-8e50-764be14e4ad9 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.779399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.779540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83774ae2-8ca3-4c84-8e50-764be14e4ad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.779902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83774ae2-8ca3-4c84-8e50-764be14e4ad9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.780008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.780248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.780534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.780641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83774ae2-8ca3-4c84-8e50-764be14e4ad9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.781277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.782397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83774ae2-8ca3-4c84-8e50-764be14e4ad9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.782920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.792815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 821/1606] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.793699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 786/1607] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:09.794627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.794906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5952c5b3-4025-48de-94c9-a17b6d751553 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.795305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5952c5b3-4025-48de-94c9-a17b6d751553 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.795574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5952c5b3-4025-48de-94c9-a17b6d751553 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.796970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 822/1608] 10.222.0.72 () {54 vars in 961 bytes} [Mon Jul 11 14:40:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:40:09.797654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5952c5b3-4025-48de-94c9-a17b6d751553 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.808643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d78d3f-cda2-411a-a46a-b4b20593d99b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.811924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5952c5b3-4025-48de-94c9-a17b6d751553 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.814147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abff79e-61f9-479b-9237-7cabc9fd1ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.815093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abff79e-61f9-479b-9237-7cabc9fd1ac8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.815380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abff79e-61f9-479b-9237-7cabc9fd1ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.815663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abff79e-61f9-479b-9237-7cabc9fd1ac8 admin admin] PATH_INFO: `/v3/projects/f6606b37accd43619058aef23f10745b/users/99a62e49e6d249c2a169ef7122e762ec/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.816418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.817284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.817563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.817840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] PATH_INFO: `/v3/projects/fd0be78cdcfc4195a0ced5868c14cda5/users/85705ea8eb6b43e18d14eb5efefab96f/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.830793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abff79e-61f9-479b-9237-7cabc9fd1ac8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6606b37accd43619058aef23f10745b, user_id=99a62e49e6d249c2a169ef7122e762ec, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.833482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abff79e-61f9-479b-9237-7cabc9fd1ac8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.834020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd0be78cdcfc4195a0ced5868c14cda5, user_id=85705ea8eb6b43e18d14eb5efefab96f, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.836833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.847492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 787/1609] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:09 2022] PUT /identity/v3/projects/f6606b37accd43619058aef23f10745b/users/99a62e49e6d249c2a169ef7122e762ec/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:09.848775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af595a40-8d29-4e23-aa6d-cc25cf89ccea None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.849128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af595a40-8d29-4e23-aa6d-cc25cf89ccea None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.849430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af595a40-8d29-4e23-aa6d-cc25cf89ccea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.856174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 823/1610] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:09 2022] PUT /identity/v3/projects/fd0be78cdcfc4195a0ced5868c14cda5/users/85705ea8eb6b43e18d14eb5efefab96f/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:09.856951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.870931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-af595a40-8d29-4e23-aa6d-cc25cf89ccea None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:09.872713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-af595a40-8d29-4e23-aa6d-cc25cf89ccea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:09.920844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb612533-d058-4428-9fef-bd0373363988 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:09.925596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:09.926529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:09.926819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:09.927105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] PATH_INFO: `/v3/projects/1baa6ea2e19747d1af1e1042c362895d/users/3ae56fd1172f4bf8806eefc6e8877fef/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:09.929392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:09.938404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 788/1611] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:09.941613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af595a40-8d29-4e23-aa6d-cc25cf89ccea None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.948532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1baa6ea2e19747d1af1e1042c362895d, user_id=3ae56fd1172f4bf8806eefc6e8877fef, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:09.950064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:09.952520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:09.968186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 824/1612] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:09 2022] PUT /identity/v3/projects/1baa6ea2e19747d1af1e1042c362895d/users/3ae56fd1172f4bf8806eefc6e8877fef/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:09.972644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:09.999810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af595a40-8d29-4e23-aa6d-cc25cf89ccea None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.036749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1d7d4e-ba6b-4271-b82b-e00d5a3af13f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.041380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.042278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.042600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.042880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] PATH_INFO: `/v3/projects/55c685b237f1412ba3b6abe72923a4fe/users/8de9e0c90c7d440ca8496c2205c109c8/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.058916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72709fb1-32af-4a2e-bc73-d05ec95ee71e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.060282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72709fb1-32af-4a2e-bc73-d05ec95ee71e service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.060739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72709fb1-32af-4a2e-bc73-d05ec95ee71e service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.060999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] RBAC: Authorizing `identity:create_grant(project_id=55c685b237f1412ba3b6abe72923a4fe, user_id=8de9e0c90c7d440ca8496c2205c109c8, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.061198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72709fb1-32af-4a2e-bc73-d05ec95ee71e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.063735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.078201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 825/1613] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:09 2022] PUT /identity/v3/projects/55c685b237f1412ba3b6abe72923a4fe/users/8de9e0c90c7d440ca8496c2205c109c8/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:10.079191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.133073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72709fb1-32af-4a2e-bc73-d05ec95ee71e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.134753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72709fb1-32af-4a2e-bc73-d05ec95ee71e service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.144865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 789/1614] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:10.146023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f6eb30-1401-48a7-9f48-42d156b81a3e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.146195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed92745-e1ee-487e-bb9a-4eace6d7225e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.146308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f6eb30-1401-48a7-9f48-42d156b81a3e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.146618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f6eb30-1401-48a7-9f48-42d156b81a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.151391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6709d143-e3c8-466e-a0f5-f789383f3413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.152218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6709d143-e3c8-466e-a0f5-f789383f3413 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.152446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6709d143-e3c8-466e-a0f5-f789383f3413 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.153457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6709d143-e3c8-466e-a0f5-f789383f3413 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.153929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6709d143-e3c8-466e-a0f5-f789383f3413 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.155721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6709d143-e3c8-466e-a0f5-f789383f3413 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.169777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 826/1615] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:10 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:10.170030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa49631-d49b-4a25-948d-0166682839be None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.170323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa49631-d49b-4a25-948d-0166682839be None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.170643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa49631-d49b-4a25-948d-0166682839be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.189141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ffa49631-d49b-4a25-948d-0166682839be None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:10.190486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ffa49631-d49b-4a25-948d-0166682839be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:10.191383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b5f6eb30-1401-48a7-9f48-42d156b81a3e None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:10.192613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b5f6eb30-1401-48a7-9f48-42d156b81a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:10.198940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 827/1616] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:10.200290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcf2db1-a72c-47fc-9edd-8c6372ecdcbc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.200580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcf2db1-a72c-47fc-9edd-8c6372ecdcbc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.200863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcf2db1-a72c-47fc-9edd-8c6372ecdcbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.221513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-0bcf2db1-a72c-47fc-9edd-8c6372ecdcbc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:10.223084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0bcf2db1-a72c-47fc-9edd-8c6372ecdcbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:10.232219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 828/1617] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:10.233120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcf2db1-a72c-47fc-9edd-8c6372ecdcbc None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.274506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:10.291843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 790/1618] 10.222.0.72 () {58 vars in 1058 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:10.293315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee122f-7756-4443-a3c0-9837a791d44a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.293691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee122f-7756-4443-a3c0-9837a791d44a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.293927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee122f-7756-4443-a3c0-9837a791d44a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.305205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:10.311551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcf2db1-a72c-47fc-9edd-8c6372ecdcbc None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.315839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-dfee122f-7756-4443-a3c0-9837a791d44a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:10.316470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.317063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dfee122f-7756-4443-a3c0-9837a791d44a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:10.317256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.317598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.317939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.325554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 791/1619] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:10.327508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.329355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.330242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.330976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.333803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.346619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 829/1620] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:10.347273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.359585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caadaab4-ae2e-4f59-b77c-7e9b151d435b service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.364632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.365559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.365843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.366125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] PATH_INFO: `/v3/projects/c0e363b1584848c1aecbe186cf91eec5/users/467992b53c4749858fd8b0ccd2db2393/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.382689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] MFA Rules not processed for user `8de9e0c90c7d440ca8496c2205c109c8`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:10.385660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0e363b1584848c1aecbe186cf91eec5, user_id=467992b53c4749858fd8b0ccd2db2393, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.389707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.392196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:10.412515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 830/1621] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:10 2022] PUT /identity/v3/projects/c0e363b1584848c1aecbe186cf91eec5/users/467992b53c4749858fd8b0ccd2db2393/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:10.413988 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.500601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 792/1622] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:10.501663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.504457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:10.524531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe47d5b6-2379-4051-bef4-f698a951ca70 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.526401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cccfa1a-f7f0-4962-a308-a9c046d86022 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.530486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.530877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.531317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.531461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.531606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.532352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.532751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.533132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.622154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.623851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.627631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.630046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.634794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 831/1623] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:10.635485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.645720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 793/1624] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:10.646755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.665713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6cd950-81f8-4b01-a8da-378c87c64a83 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.676759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b008de37-6e38-48c7-b7b7-8aaa2d8e2220 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.677695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b008de37-6e38-48c7-b7b7-8aaa2d8e2220 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.678015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b008de37-6e38-48c7-b7b7-8aaa2d8e2220 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.678306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b008de37-6e38-48c7-b7b7-8aaa2d8e2220 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.689478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b008de37-6e38-48c7-b7b7-8aaa2d8e2220 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.691742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b008de37-6e38-48c7-b7b7-8aaa2d8e2220 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.707538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 794/1625] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:10.709322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a87d6a-7050-4b58-9d79-f4c10461aea1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.709622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a87d6a-7050-4b58-9d79-f4c10461aea1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.710344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a87d6a-7050-4b58-9d79-f4c10461aea1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.737008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b6a87d6a-7050-4b58-9d79-f4c10461aea1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:10.738601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b6a87d6a-7050-4b58-9d79-f4c10461aea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:10.743780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d882880-7516-475d-af4e-45f832196d1c service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.751504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 795/1626] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:10.752450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.752599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a87d6a-7050-4b58-9d79-f4c10461aea1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.753977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.754479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.767432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a87d6a-7050-4b58-9d79-f4c10461aea1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.774290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:10.775411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.776983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.777363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.778227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.779188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.781647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.835732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 796/1627] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/projects => generated 373 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:10.839787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.854939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.857227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:10.872947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 832/1628] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:10 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:10.885051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:10.890084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:10.915772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea5d76-33c6-4adf-b6bb-dcb2e657b1ab service nova] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.925659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b80e9-4142-4e84-8da8-125a04b1d3f3 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:10.930999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:10.931962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:10.932393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:10.932790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:10.934947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:10.937758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:11.010495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 797/1629] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:10 2022] POST /identity/v3/users => generated 293 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:11.011674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:11.042141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac819dcc-2821-4839-b50a-b593edcbe5d8 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:11.048828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:11.049628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.050134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.050278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.063262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:11.074033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:11.079561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.080418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:11.083101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.085261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.095859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:11.104309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:11.110448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 833/1630] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:11.140284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 798/1631] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:11 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:11.144594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:11.156592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f428b95f-9f74-4e76-942f-76b742d2d852 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:11.162578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616233ba-9734-4eb5-937c-1acdbb926466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:11.163557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616233ba-9734-4eb5-937c-1acdbb926466 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.163932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616233ba-9734-4eb5-937c-1acdbb926466 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.164394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616233ba-9734-4eb5-937c-1acdbb926466 admin admin] PATH_INFO: `/v3/projects/de2826017582484b8b3a10070d7886a1/users/e5b12450e72241bc8f94abc04c883c1d/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.186240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616233ba-9734-4eb5-937c-1acdbb926466 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de2826017582484b8b3a10070d7886a1, user_id=e5b12450e72241bc8f94abc04c883c1d, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:11.189391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616233ba-9734-4eb5-937c-1acdbb926466 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:11.210050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 834/1632] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:11 2022] PUT /identity/v3/projects/de2826017582484b8b3a10070d7886a1/users/e5b12450e72241bc8f94abc04c883c1d/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:11.213877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:11.225248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bfa2ad-dfe2-47cd-82f4-482f83633ac3 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:11.268683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7642eca1-ca43-45ae-83f4-b133efbf9798 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.268990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7642eca1-ca43-45ae-83f4-b133efbf9798 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.269704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7642eca1-ca43-45ae-83f4-b133efbf9798 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.271590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 835/1633] 10.222.0.72 () {54 vars in 912 bytes} [Mon Jul 11 14:40:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:40:11.279887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0456e8-2b7a-47b5-a676-3b5917f6978b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.280270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0456e8-2b7a-47b5-a676-3b5917f6978b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.280618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0456e8-2b7a-47b5-a676-3b5917f6978b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.322513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:11.323692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.324244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.325350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.326618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:11.328566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:11.347166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 799/1634] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:11 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:11.348426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:11.352083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-bc0456e8-2b7a-47b5-a676-3b5917f6978b None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:11.354674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bc0456e8-2b7a-47b5-a676-3b5917f6978b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:11.361491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da943a02-23e2-43bc-a8b6-e5eece00d028 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:11.367226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f622e2e5-4bb3-43a4-b6dd-e250a4425165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:11.368441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f622e2e5-4bb3-43a4-b6dd-e250a4425165 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.368889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f622e2e5-4bb3-43a4-b6dd-e250a4425165 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.369326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f622e2e5-4bb3-43a4-b6dd-e250a4425165 admin admin] PATH_INFO: `/v3/projects/de2826017582484b8b3a10070d7886a1/users/e5b12450e72241bc8f94abc04c883c1d/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.372381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f622e2e5-4bb3-43a4-b6dd-e250a4425165 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de2826017582484b8b3a10070d7886a1, user_id=e5b12450e72241bc8f94abc04c883c1d, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:11.375562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f622e2e5-4bb3-43a4-b6dd-e250a4425165 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:11.400018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 800/1635] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:11 2022] PUT /identity/v3/projects/de2826017582484b8b3a10070d7886a1/users/e5b12450e72241bc8f94abc04c883c1d/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:11.403353 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0a255e-92a4-4f02-9371-da840e78cc19 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.403840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0a255e-92a4-4f02-9371-da840e78cc19 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.404245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0a255e-92a4-4f02-9371-da840e78cc19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.458383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6b0a255e-92a4-4f02-9371-da840e78cc19 None None] MFA Rules not processed for user `e5b12450e72241bc8f94abc04c883c1d`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:11.461248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6b0a255e-92a4-4f02-9371-da840e78cc19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:11.547473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 836/1636] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:40:11 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:40:11.560729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0456e8-2b7a-47b5-a676-3b5917f6978b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:11.633949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:11.653127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 801/1637] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:11 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:11.705986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0456e8-2b7a-47b5-a676-3b5917f6978b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:11.781056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:11.781828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:11.782129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:11.782476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:11.966827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:11.971313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:11.999079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 837/1638] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:11 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:12.303311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0a255e-92a4-4f02-9371-da840e78cc19 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:12.319319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0a255e-92a4-4f02-9371-da840e78cc19 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:12.331114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22a7223-36b1-4a85-881d-358032a53f42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:12.333234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a7223-36b1-4a85-881d-358032a53f42 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:12.333531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a7223-36b1-4a85-881d-358032a53f42 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:12.334457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a7223-36b1-4a85-881d-358032a53f42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:12.362726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22a7223-36b1-4a85-881d-358032a53f42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:12.368837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22a7223-36b1-4a85-881d-358032a53f42 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:12.416093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 802/1639] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:12 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:13.300599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:13.493650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8dd271-c20b-41de-b82d-095b5b1f343a None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:13.494325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8dd271-c20b-41de-b82d-095b5b1f343a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:13.494774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8dd271-c20b-41de-b82d-095b5b1f343a None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:13.496134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 803/1640] 10.222.0.72 () {54 vars in 961 bytes} [Mon Jul 11 14:40:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:40:13.508130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c2577d-5646-41ba-b4c7-d9bfc3c5b046 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:13.508463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c2577d-5646-41ba-b4c7-d9bfc3c5b046 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:13.509577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c2577d-5646-41ba-b4c7-d9bfc3c5b046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:13.522982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:13.525339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d417295-2519-4c0f-b8e7-e19e559ee563 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:13.538604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:13.540661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:13.542011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:13.542011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] PATH_INFO: `/v3/users/8de9e0c90c7d440ca8496c2205c109c8` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:13.542714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8de9e0c90c7d440ca8496c2205c109c8)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:13.550461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f8c2577d-5646-41ba-b4c7-d9bfc3c5b046 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:13.551703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f8c2577d-5646-41ba-b4c7-d9bfc3c5b046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:13.553367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:13.591931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 804/1641] 10.222.0.72 () {58 vars in 1058 bytes} [Mon Jul 11 14:40:13 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:13.599415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949c9b33-1ee0-4f8a-8ca9-72a619ede338 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:13.599757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949c9b33-1ee0-4f8a-8ca9-72a619ede338 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:13.600090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949c9b33-1ee0-4f8a-8ca9-72a619ede338 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:13.602034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 805/1642] 10.222.0.72 () {54 vars in 912 bytes} [Mon Jul 11 14:40:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 11 14:40:13.609663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cc66b4-2fcf-4fc0-bd8b-d755fc4bdcaf None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:13.609983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cc66b4-2fcf-4fc0-bd8b-d755fc4bdcaf None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:13.610238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cc66b4-2fcf-4fc0-bd8b-d755fc4bdcaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:13.628964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:13.679231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-92cc66b4-2fcf-4fc0-bd8b-d755fc4bdcaf None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:13.684732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-92cc66b4-2fcf-4fc0-bd8b-d755fc4bdcaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:13.701435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8de9e0c90c7d440ca8496c2205c109c8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:13.711112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 806/1643] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:40:13 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:13.720141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8de9e0c90c7d440ca8496c2205c109c8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:13.720990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8de9e0c90c7d440ca8496c2205c109c8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:13.727077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cc66b4-2fcf-4fc0-bd8b-d755fc4bdcaf None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:13.749817 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cc66b4-2fcf-4fc0-bd8b-d755fc4bdcaf None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:13.751273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8de9e0c90c7d440ca8496c2205c109c8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:13.760014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:13.760695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:13.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:13.761822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:13.774676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:13.789490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cb01ddd5-8b66-43a2-8a30-919b6fd1a272 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8de9e0c90c7d440ca8496c2205c109c8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:13.809680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 838/1644] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:40:13 2022] DELETE /identity/v3/users/8de9e0c90c7d440ca8496c2205c109c8 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:13.813760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:13.814121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:13.814382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:13.820340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:13.826294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:13.864755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 807/1645] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:13 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:13.865865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:13.867314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:13.982752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 839/1646] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:14.030333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:14.032889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:14.199449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69532d9-04d3-49c1-9942-c5bee6ae9f25 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:14.320155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:14.323412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:14.323860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:14.325489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:14.356904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:14.359129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:14.387975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 808/1647] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:15.991397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:15.992943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:16.006054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ede13-d248-46cf-9cc0-41bb793ce62c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:16.012320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:16.013214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:16.013503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:16.014132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] PATH_INFO: `/v3/projects/55c685b237f1412ba3b6abe72923a4fe` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:16.015054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55c685b237f1412ba3b6abe72923a4fe)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:16.016165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: WARNING py.warnings [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:40:16.016165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: warnings.warn(msg) Jul 11 14:40:16.016165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: Jul 11 14:40:16.017814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:16.092982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55c685b237f1412ba3b6abe72923a4fe', 'request_id': 'req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.106158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55c685b237f1412ba3b6abe72923a4fe', 'request_id': 'req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.117643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55c685b237f1412ba3b6abe72923a4fe', 'request_id': 'req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.148299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] The token cache is being invalidate because project 55c685b237f1412ba3b6abe72923a4fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:40:16.148613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.149460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.151644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 840/1648] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:40:15 2022] DELETE /identity/v3/projects/55c685b237f1412ba3b6abe72923a4fe => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:16.156301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:16.168648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b16a4b4-28fb-49cd-a00d-b2bf512a6222 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:16.262086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:16.263523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:16.263949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:16.264443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] PATH_INFO: `/v3/users/e5b12450e72241bc8f94abc04c883c1d` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:16.266735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5b12450e72241bc8f94abc04c883c1d)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:16.267948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: WARNING py.warnings [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 11 14:40:16.267948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: warnings.warn(msg) Jul 11 14:40:16.267948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: Jul 11 14:40:16.269556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:16.377658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:16.413487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5b12450e72241bc8f94abc04c883c1d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.435536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5b12450e72241bc8f94abc04c883c1d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.450833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5b12450e72241bc8f94abc04c883c1d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.451485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5b12450e72241bc8f94abc04c883c1d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.502944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea2484c4-50b0-4aca-8f08-16ad719e1875 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5b12450e72241bc8f94abc04c883c1d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:16.516577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 809/1649] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:40:16 2022] DELETE /identity/v3/users/e5b12450e72241bc8f94abc04c883c1d => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:18.399900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:18.401389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:18.425926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1f4eb-319a-4778-ac41-e1e3622b1bc2 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:18.480204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:18.567700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:18.568762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:18.575562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:18.575849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] PATH_INFO: `/v3/projects/de2826017582484b8b3a10070d7886a1` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:18.582175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de2826017582484b8b3a10070d7886a1)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:18.584941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:18.707160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de2826017582484b8b3a10070d7886a1', 'request_id': 'req-c1277982-b0ce-4a9c-9245-213cc7d056e4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:18.720510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de2826017582484b8b3a10070d7886a1', 'request_id': 'req-c1277982-b0ce-4a9c-9245-213cc7d056e4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:18.727957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de2826017582484b8b3a10070d7886a1', 'request_id': 'req-c1277982-b0ce-4a9c-9245-213cc7d056e4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:18.770717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] The token cache is being invalidate because project de2826017582484b8b3a10070d7886a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:40:18.772797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:18.773576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c1277982-b0ce-4a9c-9245-213cc7d056e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:18.776572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 841/1650] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:40:18 2022] DELETE /identity/v3/projects/de2826017582484b8b3a10070d7886a1 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:18.781859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95e2a3-dfec-43fb-a94f-3b8062cd2e94 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:18.782924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95e2a3-dfec-43fb-a94f-3b8062cd2e94 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:18.786474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95e2a3-dfec-43fb-a94f-3b8062cd2e94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:18.856609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8d95e2a3-dfec-43fb-a94f-3b8062cd2e94 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:18.859151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8d95e2a3-dfec-43fb-a94f-3b8062cd2e94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:19.192062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 810/1651] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 412 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:19.199544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f25f303-664a-4b08-90df-01481dd299df None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:19.199836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f25f303-664a-4b08-90df-01481dd299df None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:19.200281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f25f303-664a-4b08-90df-01481dd299df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:19.256282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5f25f303-664a-4b08-90df-01481dd299df None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:19.257538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5f25f303-664a-4b08-90df-01481dd299df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:19.291946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 842/1652] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:19.296772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95e2a3-dfec-43fb-a94f-3b8062cd2e94 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:19.309608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95e2a3-dfec-43fb-a94f-3b8062cd2e94 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:19.344916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:19.345777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:19.346858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:19.347567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:19.351115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:19.351627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:19.357128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:19.381435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 811/1653] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:19.385817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f25f303-664a-4b08-90df-01481dd299df None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:19.399291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f25f303-664a-4b08-90df-01481dd299df None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:19.408703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:19.409404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:19.409696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:19.409979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:19.413843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:19.418573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:19.490693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 843/1654] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:19 2022] POST /identity/v3/projects => generated 357 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:19.494075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:19.506359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50de2cf-9de0-47e2-95e7-7e79eb4560ae admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:19.706809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:19.707511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:19.707798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:19.708082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:19.708804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:19.716542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:19.769504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 812/1655] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:19 2022] POST /identity/v3/users => generated 279 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:19.773395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:19.788456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4eced1-594c-450a-bb02-728abef9850c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:19.794838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:19.795530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:19.795820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:19.796107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:19.797418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:19.799717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:19.827129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 844/1656] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:19 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:19.835004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:19.853040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3806adc8-0378-4ac5-b52c-6c977c7645b0 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:19.868936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d39400f-d270-4e72-b8bf-29b966fba86e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:19.869759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d39400f-d270-4e72-b8bf-29b966fba86e admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:19.869927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d39400f-d270-4e72-b8bf-29b966fba86e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:19.872646 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d39400f-d270-4e72-b8bf-29b966fba86e admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:19.877465 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d39400f-d270-4e72-b8bf-29b966fba86e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:19.879486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d39400f-d270-4e72-b8bf-29b966fba86e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:19.925728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:19.926835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 813/1657] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:19 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:19.947651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64eb009-3511-48b8-bf20-04921d569161 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:19.967176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:19.968106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:19.968385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:19.968665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] PATH_INFO: `/v3/projects/2b3dfa31fcc64f4b898f1242c30a0321/users/af28e2eff5384a1cb307e2e39b00cb12/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:20.010882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b3dfa31fcc64f4b898f1242c30a0321, user_id=af28e2eff5384a1cb307e2e39b00cb12, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:20.014219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:20.054416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 845/1658] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:19 2022] PUT /identity/v3/projects/2b3dfa31fcc64f4b898f1242c30a0321/users/af28e2eff5384a1cb307e2e39b00cb12/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:20.071039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed47801-8fa5-4bee-8428-43cddde940be None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:20.071752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed47801-8fa5-4bee-8428-43cddde940be None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:20.072432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed47801-8fa5-4bee-8428-43cddde940be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:20.184218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fed47801-8fa5-4bee-8428-43cddde940be None None] MFA Rules not processed for user `af28e2eff5384a1cb307e2e39b00cb12`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:20.185942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fed47801-8fa5-4bee-8428-43cddde940be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:20.245121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:20.465301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 814/1659] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:20 2022] POST /identity/v3/auth/tokens => generated 3224 bytes in 395 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:20.469945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:20.637861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:20.685771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b338c72c-3b84-46f7-9ec4-c46a4ceb6fcf admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:20.698864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:20.699625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:20.699915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:20.700201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:20.700922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:20.703100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:20.758458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 846/1660] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:20 2022] POST /identity/v3/projects => generated 357 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:20.762334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed47801-8fa5-4bee-8428-43cddde940be None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:20.780557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed47801-8fa5-4bee-8428-43cddde940be None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:20.981507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:20.982243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:20.982716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:20.983065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:20.983997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:20.986043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:21.031067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:21.031280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 815/1661] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:20 2022] POST /identity/v3/users => generated 285 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:21.047628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28039f60-f618-44fd-aa32-8112ce8260b4 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:21.062939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:21.063646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:21.063933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:21.064892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:21.064998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:21.066691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:21.103141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 847/1662] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:40:21.107981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:21.121968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aab6d3-8bab-41da-9d70-43da30cf8e0c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:21.150288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e10149-379f-43b2-80e6-749042a403a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:21.159329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e10149-379f-43b2-80e6-749042a403a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:21.159876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e10149-379f-43b2-80e6-749042a403a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:21.160295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e10149-379f-43b2-80e6-749042a403a5 admin admin] PATH_INFO: `/v3/projects/e2743865f59a4caa870ac3c826a76899/users/a2ba6fcf761d4c5e8351bf2c315a243d/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:21.212274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e10149-379f-43b2-80e6-749042a403a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2743865f59a4caa870ac3c826a76899, user_id=a2ba6fcf761d4c5e8351bf2c315a243d, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:21.217340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e10149-379f-43b2-80e6-749042a403a5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:21.282783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 816/1663] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:21 2022] PUT /identity/v3/projects/e2743865f59a4caa870ac3c826a76899/users/a2ba6fcf761d4c5e8351bf2c315a243d/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:40:21.286698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:21.302256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbca694-07d2-477b-b02c-bac11769bbe9 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:21.378150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb1510-9684-496b-832f-0305494d61c9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:21.378512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb1510-9684-496b-832f-0305494d61c9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:21.379112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb1510-9684-496b-832f-0305494d61c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:21.467424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-dffb1510-9684-496b-832f-0305494d61c9 None None] MFA Rules not processed for user `fa478fb4b41e49c5b1f9291f925cca23`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:21.473680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dffb1510-9684-496b-832f-0305494d61c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:21.568580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:21.569288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:21.569577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:21.571930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:21.595311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:21.596807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:21.705749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 848/1664] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 419 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:21.714811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:21.775460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d134411-7887-4e9e-9113-f4c913b79944 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:21.793489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4f096e-afe6-4c28-b864-39a231288e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:21.794621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4f096e-afe6-4c28-b864-39a231288e7e admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:21.795063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4f096e-afe6-4c28-b864-39a231288e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:21.795484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4f096e-afe6-4c28-b864-39a231288e7e admin admin] PATH_INFO: `/v3/projects/e2743865f59a4caa870ac3c826a76899/users/a2ba6fcf761d4c5e8351bf2c315a243d/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:21.797795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4f096e-afe6-4c28-b864-39a231288e7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2743865f59a4caa870ac3c826a76899, user_id=a2ba6fcf761d4c5e8351bf2c315a243d, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:21.809092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4f096e-afe6-4c28-b864-39a231288e7e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:21.845944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 817/1665] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 469 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:21.869587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:21.870314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:21.871158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:21.898524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 849/1666] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:21 2022] PUT /identity/v3/projects/e2743865f59a4caa870ac3c826a76899/users/a2ba6fcf761d4c5e8351bf2c315a243d/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:21.898714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:21.914622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:21.915238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:21.915910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:21.925508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] MFA Rules not processed for user `fa478fb4b41e49c5b1f9291f925cca23`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:21.929385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:22.025202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] MFA Rules not processed for user `a2ba6fcf761d4c5e8351bf2c315a243d`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:22.027949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:22.155541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 818/1667] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:40:22.163256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:22.165482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:22.294957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 850/1668] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:21 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 387 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:22.322367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:22.325875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:22.363797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe050c52-74e5-4be2-8372-4ee2dfa141f3 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:22.374697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:22.375381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:22.375669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:22.375951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:22.396782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:22.398453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:22.422289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 819/1669] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:22 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:22.425431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:22.499184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0ea107-a87c-4b06-93e9-564c57f7ec30 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:22.514100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7494b99-4cdf-4c5e-b99a-04df2b4b5a34 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:22.515059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7494b99-4cdf-4c5e-b99a-04df2b4b5a34 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:22.515522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7494b99-4cdf-4c5e-b99a-04df2b4b5a34 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:22.515945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7494b99-4cdf-4c5e-b99a-04df2b4b5a34 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:22.536334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7494b99-4cdf-4c5e-b99a-04df2b4b5a34 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:22.538044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7494b99-4cdf-4c5e-b99a-04df2b4b5a34 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:22.584018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 851/1670] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:22 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:22.799182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:22.803117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:22.821780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13d52fb-dc62-4b38-b143-7d2cb0ec014c service nova] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:22.828015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6763cbb-4a41-4240-bba7-d0545793daf3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:22.828904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6763cbb-4a41-4240-bba7-d0545793daf3 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:22.829325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6763cbb-4a41-4240-bba7-d0545793daf3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:22.830362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6763cbb-4a41-4240-bba7-d0545793daf3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:22.924626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6763cbb-4a41-4240-bba7-d0545793daf3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:22.928189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6763cbb-4a41-4240-bba7-d0545793daf3 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:22.969938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 820/1671] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:22 2022] GET /identity/v3/auth/tokens => generated 3224 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:23.712041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3fd77e-6d09-44a7-8dfa-790f82c78bde None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:23.712534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3fd77e-6d09-44a7-8dfa-790f82c78bde None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:23.712609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3fd77e-6d09-44a7-8dfa-790f82c78bde None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:23.714293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 852/1672] 10.222.0.72 () {54 vars in 912 bytes} [Mon Jul 11 14:40:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:40:23.727369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:23.727917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:23.727917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:23.827397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] MFA Rules not processed for user `ac5254fb5eff4108be9ac3992e05a99c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:23.833362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:23.895313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:23.983480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 821/1673] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:40:23 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:24.288955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3fd77e-6d09-44a7-8dfa-790f82c78bde None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:24.291199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5b3fd77e-6d09-44a7-8dfa-790f82c78bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:24.347145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3fd77e-6d09-44a7-8dfa-790f82c78bde None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:24.373397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea6898c-24ba-471c-8510-ab75c501d988 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:24.374145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6898c-24ba-471c-8510-ab75c501d988 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:24.374465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6898c-24ba-471c-8510-ab75c501d988 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:24.376694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6898c-24ba-471c-8510-ab75c501d988 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:24.411202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea6898c-24ba-471c-8510-ab75c501d988 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:24.414639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea6898c-24ba-471c-8510-ab75c501d988 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:24.447345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 853/1674] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:24 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:24.713827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:24.716037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:24.966580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c44083-a3f9-4559-af34-0473138ede69 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:24.984315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:24.985227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:24.991199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:24.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] PATH_INFO: `/v3/users/072255e7ae5147cb9a1540fd12f242da` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:24.994982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] RBAC: Authorizing `identity:delete_user(user_id=072255e7ae5147cb9a1540fd12f242da)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:25.000530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:25.108429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '072255e7ae5147cb9a1540fd12f242da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:25.121393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '072255e7ae5147cb9a1540fd12f242da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:25.135811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '072255e7ae5147cb9a1540fd12f242da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:25.137228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '072255e7ae5147cb9a1540fd12f242da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:25.170854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '072255e7ae5147cb9a1540fd12f242da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:25.188268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 822/1675] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:40:24 2022] DELETE /identity/v3/users/072255e7ae5147cb9a1540fd12f242da => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:40:25.191334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc0902-d6e2-4775-ae2d-40738bca1e05 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:25.191850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc0902-d6e2-4775-ae2d-40738bca1e05 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:25.192215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc0902-d6e2-4775-ae2d-40738bca1e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:25.271115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d0bc0902-d6e2-4775-ae2d-40738bca1e05 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:25.276037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d0bc0902-d6e2-4775-ae2d-40738bca1e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:25.509264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 854/1676] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:25.701162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:25.702784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:25.759843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429ade11-8580-4f6a-a78e-019c2413bde5 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:25.760136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429ade11-8580-4f6a-a78e-019c2413bde5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:25.760420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429ade11-8580-4f6a-a78e-019c2413bde5 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:25.762719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 855/1677] 10.222.0.72 () {54 vars in 960 bytes} [Mon Jul 11 14:40:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:40:25.768011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f61617a-757d-4455-9a26-ffdbb36bf381 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:25.768303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f61617a-757d-4455-9a26-ffdbb36bf381 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:25.768588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f61617a-757d-4455-9a26-ffdbb36bf381 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:25.806093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9f61617a-757d-4455-9a26-ffdbb36bf381 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:25.807605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9f61617a-757d-4455-9a26-ffdbb36bf381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:25.843321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e43632-6571-4dbd-8344-1d9f6b2a5577 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:25.846767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:25.858728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:25.859301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:25.859792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:25.867137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:25.873279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:25.875297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:25.898948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 823/1678] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:25.997011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 856/1679] 10.222.0.72 () {58 vars in 1057 bytes} [Mon Jul 11 14:40:25 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:40:26.002565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:26.021813 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e6f17f-90e9-4f06-9abb-01bafa985834 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:26.032997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766ced95-cd12-4621-aa30-73f2357e6427 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:26.035162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766ced95-cd12-4621-aa30-73f2357e6427 service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:26.035477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766ced95-cd12-4621-aa30-73f2357e6427 service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:26.036359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766ced95-cd12-4621-aa30-73f2357e6427 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:26.043673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-766ced95-cd12-4621-aa30-73f2357e6427 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:26.080245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766ced95-cd12-4621-aa30-73f2357e6427 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:26.096813 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766ced95-cd12-4621-aa30-73f2357e6427 service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:26.134983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 824/1680] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:26 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:26.213898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:26.214874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:26.215837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:26.266788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] MFA Rules not processed for user `fa478fb4b41e49c5b1f9291f925cca23`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:26.268360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:26.472381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 857/1681] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:26.608185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc08821-30af-449d-8233-ea0ec5d8e6bc None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:26.608488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc08821-30af-449d-8233-ea0ec5d8e6bc None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:26.608856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc08821-30af-449d-8233-ea0ec5d8e6bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:26.727490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3fc08821-30af-449d-8233-ea0ec5d8e6bc None None] MFA Rules not processed for user `99a62e49e6d249c2a169ef7122e762ec`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:26.728759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3fc08821-30af-449d-8233-ea0ec5d8e6bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:26.865833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 825/1682] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:26 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:26.890521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:26.892517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:26.904489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:27.168350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63594024-41b0-48fd-978f-9a5e4a119318 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:27.184597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:27.185551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:27.187432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:27.187913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:27.211826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:27.213775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:27.245196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 858/1683] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:26 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:27.556739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7548b31f-f421-4be6-9a16-d1ec2ccffa86 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:27.557653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7548b31f-f421-4be6-9a16-d1ec2ccffa86 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:27.558162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7548b31f-f421-4be6-9a16-d1ec2ccffa86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:27.621919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:27.624678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:27.650104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7548b31f-f421-4be6-9a16-d1ec2ccffa86 None None] MFA Rules not processed for user `3ae56fd1172f4bf8806eefc6e8877fef`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:27.651162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7548b31f-f421-4be6-9a16-d1ec2ccffa86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:27.666742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0289f35-7e8c-4564-b091-f53c5d3da9a2 service nova] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:27.689284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:27.692255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:27.701697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:27.703011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] PATH_INFO: `/v3/projects/bf9d11f0b33a402f8584b4a158b3b41b` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:27.708656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf9d11f0b33a402f8584b4a158b3b41b)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:27.725057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:27.841797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 826/1684] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:27.858236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7548b31f-f421-4be6-9a16-d1ec2ccffa86 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:27.885881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7548b31f-f421-4be6-9a16-d1ec2ccffa86 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:27.903184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bf124c-3aac-4d12-a9e8-3a239f90de9b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:27.904228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bf124c-3aac-4d12-a9e8-3a239f90de9b service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:27.904699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bf124c-3aac-4d12-a9e8-3a239f90de9b service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:27.905159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bf124c-3aac-4d12-a9e8-3a239f90de9b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:27.935135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bf124c-3aac-4d12-a9e8-3a239f90de9b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:27.937502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bf124c-3aac-4d12-a9e8-3a239f90de9b service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:27.959785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 827/1685] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:27.990511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf9d11f0b33a402f8584b4a158b3b41b', 'request_id': 'req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:28.027013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf9d11f0b33a402f8584b4a158b3b41b', 'request_id': 'req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:28.041195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf9d11f0b33a402f8584b4a158b3b41b', 'request_id': 'req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:28.100902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] The token cache is being invalidate because project bf9d11f0b33a402f8584b4a158b3b41b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:40:28.101287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:28.102791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df5f990c-f7be-4ab3-86a0-c03cbc65bc91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:28.113195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 859/1686] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:40:27 2022] DELETE /identity/v3/projects/bf9d11f0b33a402f8584b4a158b3b41b => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:40:28.115201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3a9153-d92b-4b7e-b448-63221fb7a71d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:28.115723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3a9153-d92b-4b7e-b448-63221fb7a71d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:28.116191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3a9153-d92b-4b7e-b448-63221fb7a71d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:28.157534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3c3a9153-d92b-4b7e-b448-63221fb7a71d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:28.159291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3c3a9153-d92b-4b7e-b448-63221fb7a71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:28.271877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:28.327034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793c51c4-0772-4aa0-8a0b-6df346343b90 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:28.327991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793c51c4-0772-4aa0-8a0b-6df346343b90 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:28.328488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793c51c4-0772-4aa0-8a0b-6df346343b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:28.331534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 828/1687] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:28.362167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-793c51c4-0772-4aa0-8a0b-6df346343b90 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:28.364426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-793c51c4-0772-4aa0-8a0b-6df346343b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:28.392458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3a9153-d92b-4b7e-b448-63221fb7a71d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:28.393290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 860/1688] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:28.407288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3a9153-d92b-4b7e-b448-63221fb7a71d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:28.425459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:28.426521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:28.427020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:28.427491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:28.428583 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:28.429093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:28.431698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:28.477212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793c51c4-0772-4aa0-8a0b-6df346343b90 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:28.487565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 829/1689] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:28.491899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793c51c4-0772-4aa0-8a0b-6df346343b90 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:28.509156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:28.510012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:28.510394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:28.510869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:28.511850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:28.515816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:28.578508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 861/1690] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:28 2022] POST /identity/v3/projects => generated 377 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:28.579145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:28.627383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39213427-cfaf-4658-af0e-079339d3a499 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:29.055576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:29.057979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:29.059949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:29.067420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:29.069621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:29.093429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:29.229840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 830/1691] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:28 2022] POST /identity/v3/users => generated 289 bytes in 651 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:29.234337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:29.261038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0707876e-8b5b-4854-a3a3-e3356e7d2bd5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:29.320364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:29.321687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:29.322613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:29.323191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:29.325069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:29.329461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:29.346843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 862/1692] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:29 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:29.348773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:29.379513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc51cb0-dda7-476d-be75-5ae3275fbe8e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:29.450657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28a4a1a-b310-4c34-8f3b-7f0d1c2b2fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:29.451680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a4a1a-b310-4c34-8f3b-7f0d1c2b2fc2 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:29.461395 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a4a1a-b310-4c34-8f3b-7f0d1c2b2fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:29.463483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a4a1a-b310-4c34-8f3b-7f0d1c2b2fc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:29.471562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28a4a1a-b310-4c34-8f3b-7f0d1c2b2fc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:29.485080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28a4a1a-b310-4c34-8f3b-7f0d1c2b2fc2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:29.549631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:29.554961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 831/1693] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:29 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:29.660554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37353d5c-7db5-4cde-9ca8-6b28d6328aa2 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:29.697074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:29.699365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:29.705629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:29.711283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] PATH_INFO: `/v3/projects/73deedca9d7c43638e1d19c7c5bfcade/users/25f037564f41411f8aed2385ae975fca/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:29.858817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=73deedca9d7c43638e1d19c7c5bfcade, user_id=25f037564f41411f8aed2385ae975fca, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:29.904240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:29.995272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 863/1694] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:29 2022] PUT /identity/v3/projects/73deedca9d7c43638e1d19c7c5bfcade/users/25f037564f41411f8aed2385ae975fca/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:30.002703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc793d-cc3e-4e60-a799-7ae9bf99fa43 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:30.003665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc793d-cc3e-4e60-a799-7ae9bf99fa43 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:30.004106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc793d-cc3e-4e60-a799-7ae9bf99fa43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:30.133500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-77bc793d-cc3e-4e60-a799-7ae9bf99fa43 None None] MFA Rules not processed for user `25f037564f41411f8aed2385ae975fca`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:30.136098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-77bc793d-cc3e-4e60-a799-7ae9bf99fa43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:30.317150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 832/1695] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:30 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:30.318714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:30.331011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:30.409756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776852c8-2a62-43f7-a7dd-753463d8a36d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:30.410983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776852c8-2a62-43f7-a7dd-753463d8a36d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:30.411559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776852c8-2a62-43f7-a7dd-753463d8a36d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:30.560393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc7eeb-daf5-4671-b5e1-04a58e11ed6b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:30.576922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-776852c8-2a62-43f7-a7dd-753463d8a36d None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:30.579372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-776852c8-2a62-43f7-a7dd-753463d8a36d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:30.582824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:30.583922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:30.584631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:30.585029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:30.585900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:30.588648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:30.618946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:30.655770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 864/1696] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:30 2022] POST /identity/v3/projects => generated 381 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:30.663178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:30.762383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 833/1697] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:40:30 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:30.849315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa62caa1-edab-4b33-a882-f45a7283cebc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:30.864950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:30.868244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:30.868631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:30.868936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:30.869697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:30.878716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:30.927173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 865/1698] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:30 2022] POST /identity/v3/users => generated 297 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:30.931389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776852c8-2a62-43f7-a7dd-753463d8a36d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:30.953060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776852c8-2a62-43f7-a7dd-753463d8a36d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:30.968708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:30.969919 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:30.971080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:30.974149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:30.977105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:30.988650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:31.033953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 834/1699] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:30 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:31.038333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:31.061666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f63a68-f775-4807-bb38-facab1a41bb0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:31.084977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:31.090742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:31.090742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:31.090962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] PATH_INFO: `/v3/projects/755c0e0d3ab847989d89b5d78dcabc8c/users/a2e59b23bd254abb89e68fcde0df8d8c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:31.163525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:31.165619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:31.182264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=755c0e0d3ab847989d89b5d78dcabc8c, user_id=a2e59b23bd254abb89e68fcde0df8d8c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:31.197524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:31.300562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 866/1700] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:31 2022] PUT /identity/v3/projects/755c0e0d3ab847989d89b5d78dcabc8c/users/a2e59b23bd254abb89e68fcde0df8d8c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:31.314088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:31.708827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a7e973-0e51-4572-a15b-9d12c2596fd3 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:31.728303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:31.729224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:31.729697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:31.731226 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:31.734635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:31.749906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:31.760386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:31.782025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b170e23-e96e-4366-9e6a-0e3e5c0e794c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:31.797276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 867/1701] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:31 2022] GET /identity/v3/roles => generated 1420 bytes in 487 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:31.798078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:31.842407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919a7230-3bb0-4de2-9eab-29650aae298d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:31.870228 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681331d9-b51b-40c8-8006-959495bcaa7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:31.881320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681331d9-b51b-40c8-8006-959495bcaa7f admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:31.882561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681331d9-b51b-40c8-8006-959495bcaa7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:31.883232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681331d9-b51b-40c8-8006-959495bcaa7f admin admin] PATH_INFO: `/v3/projects/755c0e0d3ab847989d89b5d78dcabc8c/users/a2e59b23bd254abb89e68fcde0df8d8c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:31.891380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681331d9-b51b-40c8-8006-959495bcaa7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=755c0e0d3ab847989d89b5d78dcabc8c, user_id=a2e59b23bd254abb89e68fcde0df8d8c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:31.907362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681331d9-b51b-40c8-8006-959495bcaa7f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:31.910835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3021bbc-32dc-4396-ad67-9ac5e0e5a264 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:31.911765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3021bbc-32dc-4396-ad67-9ac5e0e5a264 service swift] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:31.912229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3021bbc-32dc-4396-ad67-9ac5e0e5a264 service swift] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:31.912721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3021bbc-32dc-4396-ad67-9ac5e0e5a264 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:31.935911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 868/1702] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:31 2022] PUT /identity/v3/projects/755c0e0d3ab847989d89b5d78dcabc8c/users/a2e59b23bd254abb89e68fcde0df8d8c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:31.938878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa55a890-060b-40d1-a334-17ea4569028b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:31.939326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa55a890-060b-40d1-a334-17ea4569028b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:31.941960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa55a890-060b-40d1-a334-17ea4569028b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:32.030491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fa55a890-060b-40d1-a334-17ea4569028b None None] MFA Rules not processed for user `85705ea8eb6b43e18d14eb5efefab96f`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:32.032631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fa55a890-060b-40d1-a334-17ea4569028b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:32.053758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:32.144965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3021bbc-32dc-4396-ad67-9ac5e0e5a264 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:32.146640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3021bbc-32dc-4396-ad67-9ac5e0e5a264 service swift] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:32.167774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 835/1703] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:40:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 1004 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 11 14:40:32.169525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:32.169525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:32.169810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:32.227532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 869/1704] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:31 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:32.229106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:32.229452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:32.229741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:32.303129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] MFA Rules not processed for user `a2e59b23bd254abb89e68fcde0df8d8c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:32.307936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:32.345185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] MFA Rules not processed for user `85705ea8eb6b43e18d14eb5efefab96f`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:32.346650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:32.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 870/1705] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:32 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:32.388012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:32.392430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:32.585039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b4c9ad-8ffb-4b8c-bdef-4cfd1e03403b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:32.604170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c745177-7e89-4451-8622-19fb4b884216 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:32.607475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c745177-7e89-4451-8622-19fb4b884216 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:32.610455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c745177-7e89-4451-8622-19fb4b884216 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:32.611324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c745177-7e89-4451-8622-19fb4b884216 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:32.633997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c745177-7e89-4451-8622-19fb4b884216 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:32.638040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c745177-7e89-4451-8622-19fb4b884216 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:32.671806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 836/1706] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:32 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 503 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:32.679843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 871/1707] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:32 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:32.697349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:32.697615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:32.701131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:32.870848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] MFA Rules not processed for user `467992b53c4749858fd8b0ccd2db2393`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:32.872145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:32.951896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:32.953373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:33.095709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:33.191028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cbce5f-302d-48f8-8a1b-291cb7806e95 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:33.192700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 872/1708] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:32 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 512 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:33.203888 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:33.206315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:33.218982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:33.221991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:33.224127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:33.225298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:33.263951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:33.265460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:33.272726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7f140-ad78-42ae-8d46-59d696b1fd76 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:33.301071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c947bb-98cc-4662-a4b4-fe257a41b8ea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:33.304520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c947bb-98cc-4662-a4b4-fe257a41b8ea service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:33.304831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c947bb-98cc-4662-a4b4-fe257a41b8ea service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:33.305138 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c947bb-98cc-4662-a4b4-fe257a41b8ea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:33.314382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 837/1709] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:32 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:33.332395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c947bb-98cc-4662-a4b4-fe257a41b8ea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:33.334038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c947bb-98cc-4662-a4b4-fe257a41b8ea service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:33.375052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 873/1710] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:33 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:33.540122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:33.541579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:33.568417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedad4a5-1fc2-4f92-9a77-8eb6e3395f7a service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:33.582035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:33.582745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:33.583221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:33.583311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:33.775002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:33.788383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:33.794081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:33.834567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 838/1711] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:33 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:36.333581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5616e72-b078-459d-9256-a89c238b7d35 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:36.334185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5616e72-b078-459d-9256-a89c238b7d35 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:36.334185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5616e72-b078-459d-9256-a89c238b7d35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:36.393840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b5616e72-b078-459d-9256-a89c238b7d35 None None] MFA Rules not processed for user `ac5254fb5eff4108be9ac3992e05a99c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:36.405416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b5616e72-b078-459d-9256-a89c238b7d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:36.607027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 874/1712] 10.222.0.36 () {58 vars in 1015 bytes} [Mon Jul 11 14:40:36 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:36.729022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:36.730741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:36.766917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176c5c94-496a-4d70-93fd-55ee274996e6 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:36.789446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5372d59-b31e-46c1-9a3d-31ce046df4f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:36.790608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5372d59-b31e-46c1-9a3d-31ce046df4f5 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:36.796644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5372d59-b31e-46c1-9a3d-31ce046df4f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:36.802324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5372d59-b31e-46c1-9a3d-31ce046df4f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:36.875552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5372d59-b31e-46c1-9a3d-31ce046df4f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:36.881950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5372d59-b31e-46c1-9a3d-31ce046df4f5 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:36.973094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 839/1713] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:36 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:39.165089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b58ff7-be5b-48a8-be13-d18dbc7fa918 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:39.165708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b58ff7-be5b-48a8-be13-d18dbc7fa918 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:39.165708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b58ff7-be5b-48a8-be13-d18dbc7fa918 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:39.255802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-60b58ff7-be5b-48a8-be13-d18dbc7fa918 None None] MFA Rules not processed for user `3ae56fd1172f4bf8806eefc6e8877fef`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:39.265345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-60b58ff7-be5b-48a8-be13-d18dbc7fa918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:39.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 875/1714] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:41.643908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccff5f5-fa72-4d8e-a0fd-fbf8de8032ba None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:41.646336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccff5f5-fa72-4d8e-a0fd-fbf8de8032ba None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:41.646973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccff5f5-fa72-4d8e-a0fd-fbf8de8032ba None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:41.649298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 840/1715] 10.222.0.72 () {54 vars in 912 bytes} [Mon Jul 11 14:40:41 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:40:41.655509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9ec2fe-8835-4a01-be86-069d841be424 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:41.656032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9ec2fe-8835-4a01-be86-069d841be424 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:41.656140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9ec2fe-8835-4a01-be86-069d841be424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:41.686799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-6b9ec2fe-8835-4a01-be86-069d841be424 None None] MFA Rules not processed for user `ac5254fb5eff4108be9ac3992e05a99c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:41.693343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6b9ec2fe-8835-4a01-be86-069d841be424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:41.704085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 876/1716] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:40:41 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:41.743276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccff5f5-fa72-4d8e-a0fd-fbf8de8032ba None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:41.751321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1ccff5f5-fa72-4d8e-a0fd-fbf8de8032ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:41.781137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccff5f5-fa72-4d8e-a0fd-fbf8de8032ba None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:41.800208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe74936-4bce-48b0-9637-7dc1ce166c84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:41.806964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe74936-4bce-48b0-9637-7dc1ce166c84 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:41.808903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe74936-4bce-48b0-9637-7dc1ce166c84 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:41.809884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe74936-4bce-48b0-9637-7dc1ce166c84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:41.831063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe74936-4bce-48b0-9637-7dc1ce166c84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:41.832943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe74936-4bce-48b0-9637-7dc1ce166c84 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:41.861625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 841/1717] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:41 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:42.054805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f599c-8ad1-4aaa-b3db-d26a88751cbc None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:42.055187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f599c-8ad1-4aaa-b3db-d26a88751cbc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:42.055582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f599c-8ad1-4aaa-b3db-d26a88751cbc None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:42.057546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 877/1718] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:40:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:40:42.078383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17286bec-ad86-4df7-a16a-7d772baa3a40 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:42.081527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17286bec-ad86-4df7-a16a-7d772baa3a40 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:42.085115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17286bec-ad86-4df7-a16a-7d772baa3a40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:42.132249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-17286bec-ad86-4df7-a16a-7d772baa3a40 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:42.139910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-17286bec-ad86-4df7-a16a-7d772baa3a40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:42.175883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 842/1719] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:40:42 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:42.245813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f599c-8ad1-4aaa-b3db-d26a88751cbc None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:42.279477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f599c-8ad1-4aaa-b3db-d26a88751cbc None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:42.301251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755b54b9-d5ee-4673-a0e2-6953152692b9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:42.301679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755b54b9-d5ee-4673-a0e2-6953152692b9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:42.302071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755b54b9-d5ee-4673-a0e2-6953152692b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:42.320220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2e8423-a85a-4ec7-a769-f99dccc67da0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:42.321299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e8423-a85a-4ec7-a769-f99dccc67da0 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:42.321801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e8423-a85a-4ec7-a769-f99dccc67da0 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:42.322292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e8423-a85a-4ec7-a769-f99dccc67da0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:42.336033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-755b54b9-d5ee-4673-a0e2-6953152692b9 None None] MFA Rules not processed for user `99a62e49e6d249c2a169ef7122e762ec`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:42.337871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-755b54b9-d5ee-4673-a0e2-6953152692b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:42.344231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e8423-a85a-4ec7-a769-f99dccc67da0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:42.346066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e8423-a85a-4ec7-a769-f99dccc67da0 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:42.375994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 878/1720] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:42 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:42.455554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 843/1721] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:42.972225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f6f8ef-8593-48c7-a4b7-56d675d085fd None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:42.972787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f6f8ef-8593-48c7-a4b7-56d675d085fd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:42.972787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f6f8ef-8593-48c7-a4b7-56d675d085fd None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:42.974673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 879/1722] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:40:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:40:43.003938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69110b-4ae0-4868-a73b-e0e63bda7e0c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:43.004959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69110b-4ae0-4868-a73b-e0e63bda7e0c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:43.005398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69110b-4ae0-4868-a73b-e0e63bda7e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:43.042885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2d69110b-4ae0-4868-a73b-e0e63bda7e0c None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:43.044344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2d69110b-4ae0-4868-a73b-e0e63bda7e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:43.068991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 844/1723] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:40:43 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:43.954056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f6f8ef-8593-48c7-a4b7-56d675d085fd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:43.992036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f6f8ef-8593-48c7-a4b7-56d675d085fd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:44.015336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844686c-098e-4ab1-b50b-78ffea9a2eab None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:44.015822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844686c-098e-4ab1-b50b-78ffea9a2eab None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:44.015917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844686c-098e-4ab1-b50b-78ffea9a2eab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:44.018309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a262f6-7418-4d30-be6c-4a3c60494e9e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:44.020378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a262f6-7418-4d30-be6c-4a3c60494e9e service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:44.021911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a262f6-7418-4d30-be6c-4a3c60494e9e service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:44.022473 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a262f6-7418-4d30-be6c-4a3c60494e9e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:44.047530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a262f6-7418-4d30-be6c-4a3c60494e9e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:44.049181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a262f6-7418-4d30-be6c-4a3c60494e9e service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:44.061840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6844686c-098e-4ab1-b50b-78ffea9a2eab None None] MFA Rules not processed for user `467992b53c4749858fd8b0ccd2db2393`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:44.064141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6844686c-098e-4ab1-b50b-78ffea9a2eab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:44.085635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 845/1724] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:44 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:40:44.095014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 880/1725] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:43 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:48.952047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844686c-098e-4ab1-b50b-78ffea9a2eab None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:49.150754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844686c-098e-4ab1-b50b-78ffea9a2eab None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:49.185746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:49.189187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:49.191622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:49.192557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] PATH_INFO: `/v3/users/25f037564f41411f8aed2385ae975fca` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:49.199109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25f037564f41411f8aed2385ae975fca)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:49.208018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:49.451958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25f037564f41411f8aed2385ae975fca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:49.514079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:49.521956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25f037564f41411f8aed2385ae975fca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:49.559994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25f037564f41411f8aed2385ae975fca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:49.561840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25f037564f41411f8aed2385ae975fca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:49.627274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25f037564f41411f8aed2385ae975fca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:49.651951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 846/1726] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:40:48 2022] DELETE /identity/v3/users/25f037564f41411f8aed2385ae975fca => generated 0 bytes in 697 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:49.653546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:49.654075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:49.654172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:49.695241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:49.707573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:49.778088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:49.814971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 881/1727] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:40:49.984749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:49.987628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:50.309261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451c0ac7-dac5-4254-ab49-189a5616e788 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:50.328173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42781a13-dbb9-4b44-a225-399b119875c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:50.332242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42781a13-dbb9-4b44-a225-399b119875c0 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:50.333065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42781a13-dbb9-4b44-a225-399b119875c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:50.333325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42781a13-dbb9-4b44-a225-399b119875c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:50.387499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42781a13-dbb9-4b44-a225-399b119875c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:50.388982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42781a13-dbb9-4b44-a225-399b119875c0 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:50.451387 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 847/1728] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:50.854932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:50.857379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:50.875717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d5ff38-0ef4-46fc-9fbb-0a16941f41d5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:50.880022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:50.881337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:50.881690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:50.882043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] PATH_INFO: `/v3/users/af28e2eff5384a1cb307e2e39b00cb12` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:50.883922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af28e2eff5384a1cb307e2e39b00cb12)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:50.889217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:50.964094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af28e2eff5384a1cb307e2e39b00cb12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:50.975059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af28e2eff5384a1cb307e2e39b00cb12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:50.976917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af28e2eff5384a1cb307e2e39b00cb12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:50.994747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af28e2eff5384a1cb307e2e39b00cb12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.017646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af28e2eff5384a1cb307e2e39b00cb12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.029727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 882/1729] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:40:50 2022] DELETE /identity/v3/users/af28e2eff5384a1cb307e2e39b00cb12 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:51.035778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f041b-929f-422d-9639-9081faf7eaae None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:51.036692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f041b-929f-422d-9639-9081faf7eaae None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:51.037156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f041b-929f-422d-9639-9081faf7eaae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:51.055009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:51.056734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:51.083960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-849f041b-929f-422d-9639-9081faf7eaae None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:51.089894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-849f041b-929f-422d-9639-9081faf7eaae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:51.260263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e557a0c-23cc-47ca-b0db-c76c8b6169d0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:51.278996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:51.279902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:51.280393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:51.280753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] PATH_INFO: `/v3/projects/73deedca9d7c43638e1d19c7c5bfcade` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:51.285101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73deedca9d7c43638e1d19c7c5bfcade)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:51.289291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:51.346336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 848/1730] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:51.381351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849f041b-929f-422d-9639-9081faf7eaae None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:51.382812 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-849f041b-929f-422d-9639-9081faf7eaae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:51.422014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73deedca9d7c43638e1d19c7c5bfcade', 'request_id': 'req-49897350-8a88-4398-a1e3-495fc7119a33'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.449603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73deedca9d7c43638e1d19c7c5bfcade', 'request_id': 'req-49897350-8a88-4398-a1e3-495fc7119a33'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.468531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:51.475182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73deedca9d7c43638e1d19c7c5bfcade', 'request_id': 'req-49897350-8a88-4398-a1e3-495fc7119a33'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.517214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] The token cache is being invalidate because project 73deedca9d7c43638e1d19c7c5bfcade was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:40:51.518998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.522033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.528637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 883/1731] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:40:51 2022] DELETE /identity/v3/projects/73deedca9d7c43638e1d19c7c5bfcade => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:40:51.532345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:51.536078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:51.589441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849f041b-929f-422d-9639-9081faf7eaae None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:51.705675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:51.757324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:51.758413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:51.759346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:51.759346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:51.761835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49897350-8a88-4398-a1e3-495fc7119a33 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:51.765018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:51.775107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:51.776888 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:51.777504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:51.778354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] PATH_INFO: `/v3/users/a2e59b23bd254abb89e68fcde0df8d8c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:51.781131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2e59b23bd254abb89e68fcde0df8d8c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:51.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:51.792351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:51.796154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:51.822594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 849/1732] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:51.912107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2e59b23bd254abb89e68fcde0df8d8c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.932277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2e59b23bd254abb89e68fcde0df8d8c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.933654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2e59b23bd254abb89e68fcde0df8d8c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:51.958355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2e59b23bd254abb89e68fcde0df8d8c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.000956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2e59b23bd254abb89e68fcde0df8d8c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.018310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 884/1733] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:40:51 2022] DELETE /identity/v3/users/a2e59b23bd254abb89e68fcde0df8d8c => generated 0 bytes in 484 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:52.379785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:52.381914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:52.402001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4166dd-9a39-4115-b366-1fe90b288ed9 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:52.626508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:52.628229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:52.628953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:52.629638 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] PATH_INFO: `/v3/projects/2b3dfa31fcc64f4b898f1242c30a0321` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:52.631639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b3dfa31fcc64f4b898f1242c30a0321)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:52.633938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:52.764423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b3dfa31fcc64f4b898f1242c30a0321', 'request_id': 'req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.771699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b3dfa31fcc64f4b898f1242c30a0321', 'request_id': 'req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.784022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b3dfa31fcc64f4b898f1242c30a0321', 'request_id': 'req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.802617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] The token cache is being invalidate because project 2b3dfa31fcc64f4b898f1242c30a0321 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:40:52.802981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.803922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.806072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 850/1734] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:40:52 2022] DELETE /identity/v3/projects/2b3dfa31fcc64f4b898f1242c30a0321 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:52.811336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:52.813112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:52.836366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eedc8d-24e4-47ac-955e-6201feae2b9a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:52.912246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:52.913306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:52.913594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:52.913875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] PATH_INFO: `/v3/users/a2ba6fcf761d4c5e8351bf2c315a243d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:52.915313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2ba6fcf761d4c5e8351bf2c315a243d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:52.917491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:52.933774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:52.983821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2ba6fcf761d4c5e8351bf2c315a243d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.998263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2ba6fcf761d4c5e8351bf2c315a243d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:52.998995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2ba6fcf761d4c5e8351bf2c315a243d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.008462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2ba6fcf761d4c5e8351bf2c315a243d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.025864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5f0b5768-aa75-45dc-94dc-3bc44ee05879 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2ba6fcf761d4c5e8351bf2c315a243d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.037498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 885/1735] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:40:52 2022] DELETE /identity/v3/users/a2ba6fcf761d4c5e8351bf2c315a243d => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:53.636332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:53.638014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:53.654390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe69e36b-dd34-44b7-8ccd-7cc6afc00e3f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:53.739275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:53.741226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:53.741876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:53.742352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] PATH_INFO: `/v3/projects/755c0e0d3ab847989d89b5d78dcabc8c` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:53.744629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] RBAC: Authorizing `identity:delete_project(project_id=755c0e0d3ab847989d89b5d78dcabc8c)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:53.747017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:53.765340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:53.827795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '755c0e0d3ab847989d89b5d78dcabc8c', 'request_id': 'req-b4a06ada-63ef-4315-b010-d650d5a3828c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.845118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '755c0e0d3ab847989d89b5d78dcabc8c', 'request_id': 'req-b4a06ada-63ef-4315-b010-d650d5a3828c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.853704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '755c0e0d3ab847989d89b5d78dcabc8c', 'request_id': 'req-b4a06ada-63ef-4315-b010-d650d5a3828c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.871658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] The token cache is being invalidate because project 755c0e0d3ab847989d89b5d78dcabc8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:40:53.871906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.872602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b4a06ada-63ef-4315-b010-d650d5a3828c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:53.875655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 851/1736] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:40:53 2022] DELETE /identity/v3/projects/755c0e0d3ab847989d89b5d78dcabc8c => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:53.880698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f566e79-d1e3-469c-a387-8eedbb165729 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:53.880698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f566e79-d1e3-469c-a387-8eedbb165729 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:53.881048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f566e79-d1e3-469c-a387-8eedbb165729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:53.904836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-6f566e79-d1e3-469c-a387-8eedbb165729 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:53.906644 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6f566e79-d1e3-469c-a387-8eedbb165729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:53.989660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 886/1737] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:53.993054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78f840b-b96b-4b38-a8ca-f5faeb96224a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:53.993349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78f840b-b96b-4b38-a8ca-f5faeb96224a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:53.994501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78f840b-b96b-4b38-a8ca-f5faeb96224a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.022609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e78f840b-b96b-4b38-a8ca-f5faeb96224a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:54.024081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e78f840b-b96b-4b38-a8ca-f5faeb96224a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:54.034278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 852/1738] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:54.038908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f566e79-d1e3-469c-a387-8eedbb165729 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.052643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f566e79-d1e3-469c-a387-8eedbb165729 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.059447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.060501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.060944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.061374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.064053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:54.064053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.064584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.096378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 887/1739] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:54.100561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78f840b-b96b-4b38-a8ca-f5faeb96224a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.113888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78f840b-b96b-4b38-a8ca-f5faeb96224a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.125705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.126889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.127184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.127478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.128542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.130565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.169037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 853/1740] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:54 2022] POST /identity/v3/projects => generated 363 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:54.173297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.188030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfefdc6-3a96-4ec5-ad54-10cd55c670d8 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.351417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.356758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.359821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.360649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.362907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.368414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.429535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.430207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 888/1741] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:54 2022] POST /identity/v3/users => generated 282 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:54.445154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2cc7b3-03e5-4232-b4bb-36f7d20d47ad admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.451548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.452484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.452931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.453500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.454292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.456270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.471802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 854/1742] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:54 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:54.479119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.493897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2419c85f-511e-40b5-9e40-1d340942d73a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.499683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2238bd4-1606-4327-95d0-2d605fdbbf90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.500549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2238bd4-1606-4327-95d0-2d605fdbbf90 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.500843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2238bd4-1606-4327-95d0-2d605fdbbf90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.501216 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2238bd4-1606-4327-95d0-2d605fdbbf90 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.502022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2238bd4-1606-4327-95d0-2d605fdbbf90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.503678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2238bd4-1606-4327-95d0-2d605fdbbf90 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.505455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:54.521360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 889/1743] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:54 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:40:54.525331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.541255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b849fd-70d7-4468-b194-550811efce27 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.550043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.551313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.551829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.552312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] PATH_INFO: `/v3/projects/cdf933c5a44747eba680d9420d6da997/users/f68e3905a82644e6922b834c79c62b61/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.581164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=cdf933c5a44747eba680d9420d6da997, user_id=f68e3905a82644e6922b834c79c62b61, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.583849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.608266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 855/1744] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:54 2022] PUT /identity/v3/projects/cdf933c5a44747eba680d9420d6da997/users/f68e3905a82644e6922b834c79c62b61/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:54.619810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df02586-f77d-48bc-9e6d-b0d595378778 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.621180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df02586-f77d-48bc-9e6d-b0d595378778 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.621675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df02586-f77d-48bc-9e6d-b0d595378778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.675867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-6df02586-f77d-48bc-9e6d-b0d595378778 None None] MFA Rules not processed for user `f68e3905a82644e6922b834c79c62b61`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:54.677224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6df02586-f77d-48bc-9e6d-b0d595378778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:54.715658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.717581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:54.804583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 890/1745] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:54 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:54.805551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df02586-f77d-48bc-9e6d-b0d595378778 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.808015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1005f237-5fb1-490c-a4a9-b83dad8258bf admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.813402 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.814507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.814912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.815803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] PATH_INFO: `/v3/projects/e2743865f59a4caa870ac3c826a76899` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.817598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2743865f59a4caa870ac3c826a76899)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.819015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df02586-f77d-48bc-9e6d-b0d595378778 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:54.820073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.850930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:54.851628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.851920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.852205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:54.852931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:54.855017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:54.886645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:54.905192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 891/1746] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:54 2022] POST /identity/v3/projects => generated 363 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:54.909468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:54.912159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2743865f59a4caa870ac3c826a76899', 'request_id': 'req-4c8490f1-02d0-49ea-8141-4f7cb884088d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:54.919722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2743865f59a4caa870ac3c826a76899', 'request_id': 'req-4c8490f1-02d0-49ea-8141-4f7cb884088d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:54.930551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2743865f59a4caa870ac3c826a76899', 'request_id': 'req-4c8490f1-02d0-49ea-8141-4f7cb884088d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:54.964256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] The token cache is being invalidate because project e2743865f59a4caa870ac3c826a76899 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:40:54.964599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:54.967577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c8490f1-02d0-49ea-8141-4f7cb884088d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:40:54.974035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 856/1747] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:40:54 2022] DELETE /identity/v3/projects/e2743865f59a4caa870ac3c826a76899 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:54.976372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d917777f-fad0-472d-a863-aab262a87326 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:54.976372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d917777f-fad0-472d-a863-aab262a87326 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:54.976372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d917777f-fad0-472d-a863-aab262a87326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.013573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c92a57-920a-46d2-87f4-50044287abba admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.015677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d917777f-fad0-472d-a863-aab262a87326 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:55.016978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d917777f-fad0-472d-a863-aab262a87326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:55.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.044712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 857/1748] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:55.054409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ab1779-f92a-4851-8e27-483b3a71d963 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.058197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.058557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.058925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.059720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.061757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.063608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ab1779-f92a-4851-8e27-483b3a71d963 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.064294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ab1779-f92a-4851-8e27-483b3a71d963 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.105684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b9ab1779-f92a-4851-8e27-483b3a71d963 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:55.106965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b9ab1779-f92a-4851-8e27-483b3a71d963 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:55.113174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 892/1749] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:54 2022] POST /identity/v3/users => generated 288 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:55.114219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.116817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:55.139634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 858/1750] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:55.144196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ab1779-f92a-4851-8e27-483b3a71d963 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.167754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ab1779-f92a-4851-8e27-483b3a71d963 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.170340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561995d7-db8e-48bc-959c-3e7c4fa78e1d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.174745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.176206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.176650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.177202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.178369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:40:55.178870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.181182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.188793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.189862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.189976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.190231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.192196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.193855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.196846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 859/1751] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:40:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:40:55.200900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.214237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f510c1b-a627-4ed9-8929-c3152ec6387c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.219801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.221008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.221008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.221283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.221931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.223975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.226798 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 893/1752] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:55 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:55.227755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.246582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9393a633-fcde-4933-9ff7-7f09fd8adede admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.252491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.253168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.253471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.253757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] PATH_INFO: `/v3/projects/7916069778ba4780853c737fe4c61c75/users/b50250ffd62a4addb60c4e56a9e59fef/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.269369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 860/1753] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:55 2022] POST /identity/v3/projects => generated 369 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:40:55.275891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.283879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=7916069778ba4780853c737fe4c61c75, user_id=b50250ffd62a4addb60c4e56a9e59fef, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.287394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.313210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 894/1754] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:55 2022] PUT /identity/v3/projects/7916069778ba4780853c737fe4c61c75/users/b50250ffd62a4addb60c4e56a9e59fef/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:55.332532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.403312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:55.463548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58166bdd-b19f-4c01-b293-abce506e03ff admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.464345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94cc30-fcb8-4c2f-83f5-4807f96a260b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.469377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.470070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.470649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.471161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.471966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.473828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.481352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.482074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.482361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.482663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.483364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.487067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.494903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 895/1755] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:55 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:55.499709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.512748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c587a5cc-606d-4dc4-a14e-1aeceb6a46a1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.516091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 861/1756] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:55 2022] POST /identity/v3/users => generated 285 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:55.517118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.527376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.528449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.528743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.529029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] PATH_INFO: `/v3/projects/7916069778ba4780853c737fe4c61c75/users/b50250ffd62a4addb60c4e56a9e59fef/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.531192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7916069778ba4780853c737fe4c61c75, user_id=b50250ffd62a4addb60c4e56a9e59fef, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.533223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56f621e-e8fa-4370-8fa9-7f86529d1496 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.535869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.539213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3e40d0-1138-41b9-99ce-4674ecb7cae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.540097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e40d0-1138-41b9-99ce-4674ecb7cae7 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.540391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e40d0-1138-41b9-99ce-4674ecb7cae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.540673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e40d0-1138-41b9-99ce-4674ecb7cae7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.541431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3e40d0-1138-41b9-99ce-4674ecb7cae7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.543868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3e40d0-1138-41b9-99ce-4674ecb7cae7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.558475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 896/1757] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:55 2022] PUT /identity/v3/projects/7916069778ba4780853c737fe4c61c75/users/b50250ffd62a4addb60c4e56a9e59fef/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:55.561021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 862/1758] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:55 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:55.565269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.568004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af8906-fcef-4121-9548-f5231ac93e8f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.568301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af8906-fcef-4121-9548-f5231ac93e8f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.568584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af8906-fcef-4121-9548-f5231ac93e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.598316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-00af8906-fcef-4121-9548-f5231ac93e8f None None] MFA Rules not processed for user `b50250ffd62a4addb60c4e56a9e59fef`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:55.599602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-00af8906-fcef-4121-9548-f5231ac93e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:55.648336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:55.654505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe87943-9a41-4f2b-bbf3-f335a495eac2 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.667093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.667807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.668100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.669237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.674768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.674768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.689732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 863/1759] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:55 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:55.693086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 897/1760] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:55 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:40:55.695502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af8906-fcef-4121-9548-f5231ac93e8f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.702582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.704404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:55.708529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af8906-fcef-4121-9548-f5231ac93e8f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.715420 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bc0486-8a55-42ba-9808-30287c1573ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.716477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc0486-8a55-42ba-9808-30287c1573ad admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.717114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc0486-8a55-42ba-9808-30287c1573ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.717114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc0486-8a55-42ba-9808-30287c1573ad admin admin] PATH_INFO: `/v3/projects/f7be548a2d79433d8e945dac40488424/users/afe81fcb852a4b659219b4ce2ef20abc/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.733313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bc0486-8a55-42ba-9808-30287c1573ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7be548a2d79433d8e945dac40488424, user_id=afe81fcb852a4b659219b4ce2ef20abc, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.736026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bc0486-8a55-42ba-9808-30287c1573ad admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.750537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 864/1761] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:55 2022] PUT /identity/v3/projects/f7be548a2d79433d8e945dac40488424/users/afe81fcb852a4b659219b4ce2ef20abc/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:55.760743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51914a5e-8968-4b8c-9e86-5be9de176b0c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.761039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51914a5e-8968-4b8c-9e86-5be9de176b0c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.761320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51914a5e-8968-4b8c-9e86-5be9de176b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.771358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665dc37-72ee-45e6-a801-02a5308b4bdf admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.788433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-51914a5e-8968-4b8c-9e86-5be9de176b0c None None] MFA Rules not processed for user `afe81fcb852a4b659219b4ce2ef20abc`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:55.789659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-51914a5e-8968-4b8c-9e86-5be9de176b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:55.823097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.823762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.824048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.824336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.829916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:55.857748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 865/1762] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:55 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:40:55.862302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51914a5e-8968-4b8c-9e86-5be9de176b0c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.890606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.892103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.902721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 898/1763] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:55 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:55.928731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51914a5e-8968-4b8c-9e86-5be9de176b0c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:55.938766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:55.939441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:55.939726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:55.940000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:55.940693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:55.942872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:55.974131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 866/1764] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:40:55 2022] POST /identity/v3/projects => generated 369 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:40:55.977748 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.983126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:55.989864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc8bfa2-73cc-441e-b8dc-b6bd5ed7cb69 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.016103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:56.051700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a29d209-5aed-4736-a932-55fbd35b5edb admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.055183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.055861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.056149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.056432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.058861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.059376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:56.059578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.059823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.060121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.060819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:56.062841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:56.096601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 899/1765] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:40:55 2022] POST /identity/v3/users => generated 291 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:40:56.100497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:56.115136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465bb99-9f71-4eab-b78f-4347ec12f224 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.120847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.121532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.121960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.122247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.123538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:56.125215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:56.136441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:56.137977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:56.142755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 900/1766] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:56 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:56.146461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:56.149311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 867/1767] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:55 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:40:56.164115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7459e110-1388-4887-9feb-9d84170e91f0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.169720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.170737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.171169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.171530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] PATH_INFO: `/v3/projects/277f6325754e490989a05d5d4e89526d/users/b03860f35d8741d887e2a70cfc5d16b2/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.194554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] RBAC: Authorizing `identity:create_grant(project_id=277f6325754e490989a05d5d4e89526d, user_id=b03860f35d8741d887e2a70cfc5d16b2, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:56.197861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:56.247475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 901/1768] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:56 2022] PUT /identity/v3/projects/277f6325754e490989a05d5d4e89526d/users/b03860f35d8741d887e2a70cfc5d16b2/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:40:56.252427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:56.273779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f12ce8-c449-4442-a9d1-de917cf9e32c service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.358503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73b6443-46c8-42a3-a364-a892225980c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.358503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b6443-46c8-42a3-a364-a892225980c6 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.358503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b6443-46c8-42a3-a364-a892225980c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.358894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b6443-46c8-42a3-a364-a892225980c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.360175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73b6443-46c8-42a3-a364-a892225980c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:56.361838 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73b6443-46c8-42a3-a364-a892225980c6 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:56.374326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 868/1769] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:40:56 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:40:56.378707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:56.391775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45cc3f7-215b-44fd-a9f1-b54f9af7659c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.410705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.412199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.412551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.413114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] PATH_INFO: `/v3/projects/277f6325754e490989a05d5d4e89526d/users/b03860f35d8741d887e2a70cfc5d16b2/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.416003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=277f6325754e490989a05d5d4e89526d, user_id=b03860f35d8741d887e2a70cfc5d16b2, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:56.419445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:56.436547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 902/1770] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:40:56 2022] PUT /identity/v3/projects/277f6325754e490989a05d5d4e89526d/users/b03860f35d8741d887e2a70cfc5d16b2/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:40:56.447740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e15043-d209-4bf0-91e7-976a21228625 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.448099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e15043-d209-4bf0-91e7-976a21228625 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.448945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e15043-d209-4bf0-91e7-976a21228625 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.502143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-26e15043-d209-4bf0-91e7-976a21228625 None None] MFA Rules not processed for user `b03860f35d8741d887e2a70cfc5d16b2`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:56.505621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-26e15043-d209-4bf0-91e7-976a21228625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:56.652964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 869/1771] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:40:56 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:56.668392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:56.793656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:56.810768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b900f6-9244-4c4a-bab4-ec3562d5b8b7 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.817063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c2da1a-2537-41b5-9016-34918c21888a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.818115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2da1a-2537-41b5-9016-34918c21888a service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.819848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2da1a-2537-41b5-9016-34918c21888a service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.820350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2da1a-2537-41b5-9016-34918c21888a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:56.837023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c2da1a-2537-41b5-9016-34918c21888a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:56.839747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c2da1a-2537-41b5-9016-34918c21888a service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:56.859651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 903/1772] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:56 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:56.936205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e15043-d209-4bf0-91e7-976a21228625 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:56.941818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:40:56.950041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e15043-d209-4bf0-91e7-976a21228625 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:56.955762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e310af5f-4e8b-4729-af1d-99896c0ab56d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:56.956848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e310af5f-4e8b-4729-af1d-99896c0ab56d service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:56.957275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e310af5f-4e8b-4729-af1d-99896c0ab56d service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:56.957634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e310af5f-4e8b-4729-af1d-99896c0ab56d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:57.131762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e310af5f-4e8b-4729-af1d-99896c0ab56d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:57.133439 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e310af5f-4e8b-4729-af1d-99896c0ab56d service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:57.149532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 870/1773] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:56 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:40:59.233477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b68d6-c0f9-42e4-a341-f1389691b7a6 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:59.234018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b68d6-c0f9-42e4-a341-f1389691b7a6 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:59.234183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b68d6-c0f9-42e4-a341-f1389691b7a6 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:59.235770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 904/1774] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:40:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:40:59.277032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:59.277631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:59.277907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:59.319566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:40:59.323304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:59.482117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 871/1775] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:40:59 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:40:59.661401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198b68d6-c0f9-42e4-a341-f1389691b7a6 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:40:59.663328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-198b68d6-c0f9-42e4-a341-f1389691b7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:40:59.704897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198b68d6-c0f9-42e4-a341-f1389691b7a6 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:40:59.721467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6939c39-d439-42b6-a170-96c461a5d1dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:40:59.722549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6939c39-d439-42b6-a170-96c461a5d1dd service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:40:59.723467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6939c39-d439-42b6-a170-96c461a5d1dd service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:40:59.723964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6939c39-d439-42b6-a170-96c461a5d1dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:40:59.751089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6939c39-d439-42b6-a170-96c461a5d1dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:40:59.752982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6939c39-d439-42b6-a170-96c461a5d1dd service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:40:59.772439 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 905/1776] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:40:59 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:00.435535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:00.437043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:00.544258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbc203-bb89-4acb-8ed8-00a28a114df9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:00.553882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:00.555100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:00.555557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:00.556116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] PATH_INFO: `/v3/users/f11d3a5af8f1442ab935343809610f9f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:00.557868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f11d3a5af8f1442ab935343809610f9f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:00.560034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:00.613510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:00.628903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f11d3a5af8f1442ab935343809610f9f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:00.638066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f11d3a5af8f1442ab935343809610f9f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:00.647591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f11d3a5af8f1442ab935343809610f9f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:00.648416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f11d3a5af8f1442ab935343809610f9f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:00.681270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f11d3a5af8f1442ab935343809610f9f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:00.693991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 872/1777] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:00 2022] DELETE /identity/v3/users/f11d3a5af8f1442ab935343809610f9f => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:00.698935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:00.699443 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:00.699784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:00.734762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:00.737286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:00.878370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 906/1778] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:00.911136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:00.912621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:00.987098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e9ea2c-46de-4488-adee-3863802b4e24 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:00.990860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:00.991523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:00.991797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:00.992100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:01.002208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:01.003800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:01.015530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 873/1779] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:01.355691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:01.357254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:01.371701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aeb7d2d-ead4-4936-aa5a-b4740fc65e51 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:01.380207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:01.381103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:01.381383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:01.381713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] PATH_INFO: `/v3/users/afe81fcb852a4b659219b4ce2ef20abc` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:01.385004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afe81fcb852a4b659219b4ce2ef20abc)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:01.385004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:01.399117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:01.401095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:01.422506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d61d73-8eec-4ce7-b755-8e4955683844 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:01.427374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:01.428339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:01.428519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:01.428799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] PATH_INFO: `/v3/projects/7eba23022dd5473a963ee8125858ccfa` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:01.430161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eba23022dd5473a963ee8125858ccfa)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:01.433192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:01.460802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afe81fcb852a4b659219b4ce2ef20abc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.471674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afe81fcb852a4b659219b4ce2ef20abc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.472362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afe81fcb852a4b659219b4ce2ef20abc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.483242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afe81fcb852a4b659219b4ce2ef20abc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.500521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eba23022dd5473a963ee8125858ccfa', 'request_id': 'req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.505484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c06cd85-f3b6-455c-b4c6-af287cc1eca5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afe81fcb852a4b659219b4ce2ef20abc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.506615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eba23022dd5473a963ee8125858ccfa', 'request_id': 'req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.516902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 907/1780] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:01 2022] DELETE /identity/v3/users/afe81fcb852a4b659219b4ce2ef20abc => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:01.520832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eba23022dd5473a963ee8125858ccfa', 'request_id': 'req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.521512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:01.524659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:01.524958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:01.534272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] The token cache is being invalidate because project 7eba23022dd5473a963ee8125858ccfa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:01.534850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.536004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 874/1781] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:01 2022] DELETE /identity/v3/projects/7eba23022dd5473a963ee8125858ccfa => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:01.542777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:01.544189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:01.552489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:01.554545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:01.590934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:01.607599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:01.622314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c614e9f-eb7d-4db5-8e83-9f1ff3f040fa admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:01.628209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:01.629712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:01.629712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:01.629921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 908/1782] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:01.629998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] PATH_INFO: `/v3/users/ac7147b8e31841f8915cfaf4fedb07e3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:01.631049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac7147b8e31841f8915cfaf4fedb07e3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:01.633141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:01.641144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:01.642621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:01.707424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac7147b8e31841f8915cfaf4fedb07e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.716738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac7147b8e31841f8915cfaf4fedb07e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.727708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac7147b8e31841f8915cfaf4fedb07e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.728351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac7147b8e31841f8915cfaf4fedb07e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.728707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f403ad7-7180-4783-a399-161ac5060024 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:01.735187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:01.735937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:01.736395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:01.736850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:01.744257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac7147b8e31841f8915cfaf4fedb07e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:01.749421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:01.752925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 875/1783] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:01 2022] DELETE /identity/v3/users/ac7147b8e31841f8915cfaf4fedb07e3 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:01.755976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:01.768995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 909/1784] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:02.146384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.148444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:02.233794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.236247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:02.264405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032e5bb0-f94a-4fbb-856b-5b863e96b229 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.273754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b63ec82-13a7-4f4d-ab37-d15fe3a563b4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.274915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.275817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.276177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.276462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] PATH_INFO: `/v3/projects/f7be548a2d79433d8e945dac40488424` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.277696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7be548a2d79433d8e945dac40488424)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.279284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.280174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.280464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.280790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] PATH_INFO: `/v3/projects/7e9dbf4ac1d549828059d61faabd4ccc` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.282230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e9dbf4ac1d549828059d61faabd4ccc)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.284371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.334651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7be548a2d79433d8e945dac40488424', 'request_id': 'req-f6a0b961-e600-4d85-ace7-4031d94cc140'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.340649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e9dbf4ac1d549828059d61faabd4ccc', 'request_id': 'req-94b26aad-59bd-4242-8c5b-87677382cdb9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.342979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7be548a2d79433d8e945dac40488424', 'request_id': 'req-f6a0b961-e600-4d85-ace7-4031d94cc140'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.346971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e9dbf4ac1d549828059d61faabd4ccc', 'request_id': 'req-94b26aad-59bd-4242-8c5b-87677382cdb9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.348964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:02.349147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7be548a2d79433d8e945dac40488424', 'request_id': 'req-f6a0b961-e600-4d85-ace7-4031d94cc140'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.356835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e9dbf4ac1d549828059d61faabd4ccc', 'request_id': 'req-94b26aad-59bd-4242-8c5b-87677382cdb9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.361935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] The token cache is being invalidate because project f7be548a2d79433d8e945dac40488424 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:02.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.362949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.364553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 910/1785] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:02 2022] DELETE /identity/v3/projects/f7be548a2d79433d8e945dac40488424 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:02.368585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.370531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:02.373757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] The token cache is being invalidate because project 7e9dbf4ac1d549828059d61faabd4ccc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:02.373757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.374178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-94b26aad-59bd-4242-8c5b-87677382cdb9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.378529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 876/1786] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:02 2022] DELETE /identity/v3/projects/7e9dbf4ac1d549828059d61faabd4ccc => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:02.380806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ec5a8-f663-48bc-ac8f-dd667967b724 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.381112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ec5a8-f663-48bc-ac8f-dd667967b724 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.381398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ec5a8-f663-48bc-ac8f-dd667967b724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.401934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3e0ec5a8-f663-48bc-ac8f-dd667967b724 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:02.403734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3e0ec5a8-f663-48bc-ac8f-dd667967b724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:02.450182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0b961-e600-4d85-ace7-4031d94cc140 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.472227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:02.491009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 877/1787] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:02.500429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b8ac33-f321-4a0d-8f80-8551e3edb09c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.500731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b8ac33-f321-4a0d-8f80-8551e3edb09c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.501278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b8ac33-f321-4a0d-8f80-8551e3edb09c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.521307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.521541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-47b8ac33-f321-4a0d-8f80-8551e3edb09c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:02.522679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.522990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-47b8ac33-f321-4a0d-8f80-8551e3edb09c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:02.523100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.523235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] PATH_INFO: `/v3/users/b03860f35d8741d887e2a70cfc5d16b2` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.524486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b03860f35d8741d887e2a70cfc5d16b2)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.526804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.535309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 878/1788] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:02.539370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b8ac33-f321-4a0d-8f80-8551e3edb09c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.550325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b8ac33-f321-4a0d-8f80-8551e3edb09c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.555409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.556212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.557104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.557104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.558458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:02.558914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.560634 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.577100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 879/1789] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:41:02.583844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.596561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b03860f35d8741d887e2a70cfc5d16b2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.598105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e630ff-a1a8-4f81-a787-d29a8200592f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.605586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.605894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b03860f35d8741d887e2a70cfc5d16b2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.606472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.606608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b03860f35d8741d887e2a70cfc5d16b2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.606776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.607063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.607809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.609686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.619420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b03860f35d8741d887e2a70cfc5d16b2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.634086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b03860f35d8741d887e2a70cfc5d16b2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:02.640609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 911/1790] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:02 2022] DELETE /identity/v3/users/b03860f35d8741d887e2a70cfc5d16b2 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:02.643176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 880/1791] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:02 2022] POST /identity/v3/projects => generated 393 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:02.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.649170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:02.746932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71494e9a-457e-4059-a97b-223527341506 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.805175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:02.825457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.826864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.827262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.827676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.828565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.831395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.858914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 912/1792] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:02 2022] POST /identity/v3/users => generated 297 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:02.863028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.875461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfd0b37-4b45-4ab6-ade6-13667a770927 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.881374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.882197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.882596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.882944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.883891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.887436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.901268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 881/1793] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:02 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:02.906039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.920554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcde014-25c5-439f-8fb8-b464318e91aa admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.925697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea422794-73ec-461d-883b-047815f60f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.926588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea422794-73ec-461d-883b-047815f60f73 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.926879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea422794-73ec-461d-883b-047815f60f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.927164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea422794-73ec-461d-883b-047815f60f73 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.927913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea422794-73ec-461d-883b-047815f60f73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.929541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea422794-73ec-461d-883b-047815f60f73 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:02.941170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 913/1794] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:02 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:02.945410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:02.957082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92092a5-6f3c-4165-90dd-007ad44c214d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:02.967589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:02.969061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:02.969455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:02.969848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] PATH_INFO: `/v3/projects/cd47264e0cf94160ae4f6330bde0ff87/users/2d6f736ec2844a1db8fb1a0c85ba4766/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:02.991398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd47264e0cf94160ae4f6330bde0ff87, user_id=2d6f736ec2844a1db8fb1a0c85ba4766, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:02.994015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.017302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 882/1795] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:02 2022] PUT /identity/v3/projects/cd47264e0cf94160ae4f6330bde0ff87/users/2d6f736ec2844a1db8fb1a0c85ba4766/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:03.028785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dd53b5-6e52-4fe4-93a1-c43087f2eaba None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.029070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dd53b5-6e52-4fe4-93a1-c43087f2eaba None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.029354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dd53b5-6e52-4fe4-93a1-c43087f2eaba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.033877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.038163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:03.086247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a3dd53b5-6e52-4fe4-93a1-c43087f2eaba None None] MFA Rules not processed for user `2d6f736ec2844a1db8fb1a0c85ba4766`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:03.087637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a3dd53b5-6e52-4fe4-93a1-c43087f2eaba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:03.164990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 914/1796] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:03.166219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dd53b5-6e52-4fe4-93a1-c43087f2eaba None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.170302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f3cf9-6a7e-497d-a064-32762e1daed0 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.175269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.176149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.176438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.176717 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] PATH_INFO: `/v3/projects/277f6325754e490989a05d5d4e89526d` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.177960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] RBAC: Authorizing `identity:delete_project(project_id=277f6325754e490989a05d5d4e89526d)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.178559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dd53b5-6e52-4fe4-93a1-c43087f2eaba None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.179933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.185133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.186072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.186333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.186718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.187505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.189482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.194970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:INSERT=3,keystone:DELETE=1,keystone:SELECT=96 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:03.222117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 915/1797] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/projects => generated 391 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:03.226924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.246058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '277f6325754e490989a05d5d4e89526d', 'request_id': 'req-197b266e-c340-4b94-a805-1b6449b1cdec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:03.252483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '277f6325754e490989a05d5d4e89526d', 'request_id': 'req-197b266e-c340-4b94-a805-1b6449b1cdec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:03.262209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '277f6325754e490989a05d5d4e89526d', 'request_id': 'req-197b266e-c340-4b94-a805-1b6449b1cdec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:03.282569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] The token cache is being invalidate because project 277f6325754e490989a05d5d4e89526d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:03.282569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:03.282881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-197b266e-c340-4b94-a805-1b6449b1cdec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:03.287748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 883/1798] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:03 2022] DELETE /identity/v3/projects/277f6325754e490989a05d5d4e89526d => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:03.289436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6e61b4-d31b-40a5-b463-cacbf57988e1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.289736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6e61b4-d31b-40a5-b463-cacbf57988e1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.290129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6e61b4-d31b-40a5-b463-cacbf57988e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.310305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ce6e61b4-d31b-40a5-b463-cacbf57988e1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:03.312106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ce6e61b4-d31b-40a5-b463-cacbf57988e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:03.319726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585905c9-e1ff-48cf-836c-e9b094e34678 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.372513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:03.386046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 884/1799] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:03.392609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d8a55-4f79-4a77-892d-aaa5d69bbbbd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.392903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d8a55-4f79-4a77-892d-aaa5d69bbbbd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.393192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d8a55-4f79-4a77-892d-aaa5d69bbbbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.413974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.415155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.415395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.415683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.416447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.425007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-232d8a55-4f79-4a77-892d-aaa5d69bbbbd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:03.425316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.425477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-232d8a55-4f79-4a77-892d-aaa5d69bbbbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:03.440948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 885/1800] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:03.442269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232d8a55-4f79-4a77-892d-aaa5d69bbbbd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.453698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232d8a55-4f79-4a77-892d-aaa5d69bbbbd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.464353 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.465470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.465580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 916/1801] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/users => generated 302 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:03.465854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.466270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.467853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:03.468414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.469590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.470155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.471376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:03.482799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 886/1802] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:03.483541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d4ceb1-e2e1-4aee-b37f-21025882d81e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.487721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.493176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.494001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.494296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.494758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.495694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.497554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.506279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb275c12-bc14-4fe4-a0d6-6b63060e31b7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.509418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 917/1803] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:03.513393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.513990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.514517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.514805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.515113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.515987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.518602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.524366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cf9d0c-e332-4b1a-9f16-cd4fcbaa0ebb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.529725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.530659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.530943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.531222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] PATH_INFO: `/v3/projects/252a54011bd742e787a34e5c4e4cb570/users/9be0c39097c44929ac97e77d906d9d4c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.549179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=252a54011bd742e787a34e5c4e4cb570, user_id=9be0c39097c44929ac97e77d906d9d4c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.549844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 887/1804] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/projects => generated 401 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:03.552034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.554448 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.568997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 918/1805] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:03 2022] PUT /identity/v3/projects/252a54011bd742e787a34e5c4e4cb570/users/9be0c39097c44929ac97e77d906d9d4c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:03.573372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.633499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beda5315-a06e-44c8-895a-38a0284d7cef admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.638826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:03.681933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8f62a5-dcb4-409a-a112-9f88e7360de4 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.689912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.690723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.691012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.691297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.692109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.693753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.724399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 919/1806] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:03.724399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.727993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef03f174-3e0a-44b0-99bb-20a29c54fe1e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.734042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee41541e-fd0d-4f97-b795-e1d3a2433fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.734998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee41541e-fd0d-4f97-b795-e1d3a2433fc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.735288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee41541e-fd0d-4f97-b795-e1d3a2433fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.735575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee41541e-fd0d-4f97-b795-e1d3a2433fc5 admin admin] PATH_INFO: `/v3/projects/252a54011bd742e787a34e5c4e4cb570/users/9be0c39097c44929ac97e77d906d9d4c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.737638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee41541e-fd0d-4f97-b795-e1d3a2433fc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=252a54011bd742e787a34e5c4e4cb570, user_id=9be0c39097c44929ac97e77d906d9d4c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.740385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee41541e-fd0d-4f97-b795-e1d3a2433fc5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.763068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.763326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.763636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.763815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.763915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 920/1807] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:03 2022] PUT /identity/v3/projects/252a54011bd742e787a34e5c4e4cb570/users/9be0c39097c44929ac97e77d906d9d4c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:03.764681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.766415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.766958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.767075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.767483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.797124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 888/1808] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/users => generated 301 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:03.803581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.819661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] MFA Rules not processed for user `9be0c39097c44929ac97e77d906d9d4c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:03.819894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa8999b-5871-4dbd-afda-6acc2982be10 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.820916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:03.902590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 921/1809] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:03 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:03.903602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.904515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.904859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.905235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.906360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.908218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.913786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.916065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:03.923378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 889/1810] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:03.932054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:03.933537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:03.953167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067fc3e3-9df6-43cb-8ba9-3bc27a09989a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:03.961833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:03.964000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:03.964829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:03.965940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:03.967168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:03.971620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:03.988911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 890/1811] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:04.005508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.031651 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c23fb1-f39a-4f25-9610-5fb5a2caf234 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.033668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1c7467-691c-4dbd-a9a9-12412842fa7e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.039560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6f15bb-c967-4619-830d-36d92668908d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.039560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6f15bb-c967-4619-830d-36d92668908d admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.039560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6f15bb-c967-4619-830d-36d92668908d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.039560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6f15bb-c967-4619-830d-36d92668908d admin admin] PATH_INFO: `/v3/projects/28ebea9991bb48eda27cde2776d05a22/users/1eb9a16ea2e24da29c2e26a19a2e5a20/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.045036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.045825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.046217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.046625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.061090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6f15bb-c967-4619-830d-36d92668908d admin admin] RBAC: Authorizing `identity:create_grant(project_id=28ebea9991bb48eda27cde2776d05a22, user_id=1eb9a16ea2e24da29c2e26a19a2e5a20, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.066129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6f15bb-c967-4619-830d-36d92668908d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.074295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.075898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.084375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 891/1812] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:03 2022] PUT /identity/v3/projects/28ebea9991bb48eda27cde2776d05a22/users/1eb9a16ea2e24da29c2e26a19a2e5a20/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:04.095585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e88fe0-b0ab-47e9-9132-b1f7db11c7e2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.096118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e88fe0-b0ab-47e9-9132-b1f7db11c7e2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.096630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e88fe0-b0ab-47e9-9132-b1f7db11c7e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.102100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 922/1813] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:03 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:04.138750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f7e88fe0-b0ab-47e9-9132-b1f7db11c7e2 None None] MFA Rules not processed for user `1eb9a16ea2e24da29c2e26a19a2e5a20`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:04.140836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f7e88fe0-b0ab-47e9-9132-b1f7db11c7e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:04.207409 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.221281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611cc245-7c26-40d3-b529-2da06c839370 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.232801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:04.283343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 892/1814] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:04 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:04.287126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e88fe0-b0ab-47e9-9132-b1f7db11c7e2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.329402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.335451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.335751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.336534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.338119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:04.430980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e88fe0-b0ab-47e9-9132-b1f7db11c7e2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.437683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.438743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.439257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.439840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.440702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.443003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.462246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.465049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.492361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 923/1815] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:04 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:04.493569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.493773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 893/1816] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:04 2022] POST /identity/v3/projects => generated 397 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:04.523568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c22c2d1-8774-456d-b69e-a2e8ceac1428 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.607686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.608517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.608867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.609439 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.610261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.613171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.622959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:04.653986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 924/1817] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:04 2022] POST /identity/v3/users => generated 305 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:04.654898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.667645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898ce992-cb13-47ab-988b-0570a8fc968f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.672617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.673410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.673711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.673997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.674770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.676394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.690466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 894/1818] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:04 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:04.693849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.706272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e3dba-9fdf-4a8a-b36b-53627589893f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.712021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.713056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.713328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.715263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] PATH_INFO: `/v3/projects/5c69e33d94b04656908a7041cc37f11e/users/8c071c176fe646f692404ba9956d596b/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.734520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c69e33d94b04656908a7041cc37f11e, user_id=8c071c176fe646f692404ba9956d596b, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.737639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.754680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 925/1819] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:04 2022] PUT /identity/v3/projects/5c69e33d94b04656908a7041cc37f11e/users/8c071c176fe646f692404ba9956d596b/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:04.758411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.771730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd8370f-a8e9-47a4-bc72-50f9287ff380 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.851318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6e4a12-058b-4fbe-a613-d98b3b03a4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.852155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6e4a12-058b-4fbe-a613-d98b3b03a4da admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.853098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6e4a12-058b-4fbe-a613-d98b3b03a4da admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.855119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6e4a12-058b-4fbe-a613-d98b3b03a4da admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.856179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6e4a12-058b-4fbe-a613-d98b3b03a4da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.860573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6e4a12-058b-4fbe-a613-d98b3b03a4da admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.893953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 895/1820] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:04 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:04.896449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:04.912068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d9aadd-a4cc-4ba5-a86d-204cc07a59df admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:04.918609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:04.919527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.920247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.920540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] PATH_INFO: `/v3/projects/5c69e33d94b04656908a7041cc37f11e/users/8c071c176fe646f692404ba9956d596b/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.923309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c69e33d94b04656908a7041cc37f11e, user_id=8c071c176fe646f692404ba9956d596b, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:04.926002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:04.942825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 926/1821] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:04 2022] PUT /identity/v3/projects/5c69e33d94b04656908a7041cc37f11e/users/8c071c176fe646f692404ba9956d596b/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:04.955266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ac9a3-4f8f-4699-99d5-25f714d3f30b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:04.955568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ac9a3-4f8f-4699-99d5-25f714d3f30b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:04.955849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ac9a3-4f8f-4699-99d5-25f714d3f30b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:04.989892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9d5ac9a3-4f8f-4699-99d5-25f714d3f30b None None] MFA Rules not processed for user `8c071c176fe646f692404ba9956d596b`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:04.991592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9d5ac9a3-4f8f-4699-99d5-25f714d3f30b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:05.059377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:05.096168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 896/1822] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:04 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:05.108170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:05.169006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27327cd-37d7-452f-98af-9ebe04613323 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:05.172926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:05.173533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:05.173852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:05.174101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:05.184853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:05.186349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:05.197269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 927/1823] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:05 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:05.279983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5ac9a3-4f8f-4699-99d5-25f714d3f30b None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:05.291245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5ac9a3-4f8f-4699-99d5-25f714d3f30b None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:05.295109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f1a95d-adc6-4ae1-94c2-96a62bf486c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:05.295779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f1a95d-adc6-4ae1-94c2-96a62bf486c0 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:05.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f1a95d-adc6-4ae1-94c2-96a62bf486c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:05.296401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f1a95d-adc6-4ae1-94c2-96a62bf486c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:05.358780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f1a95d-adc6-4ae1-94c2-96a62bf486c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:05.360288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f1a95d-adc6-4ae1-94c2-96a62bf486c0 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:05.370502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 897/1824] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:05 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:06.802279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:06.804529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:06.945662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f48c7d-7c57-4f26-9d96-93d1ba527a45 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:06.964036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:06.965319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:06.965940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:06.966486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] PATH_INFO: `/v3/users/f68e3905a82644e6922b834c79c62b61` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:06.968177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f68e3905a82644e6922b834c79c62b61)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:06.970334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:07.051149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f68e3905a82644e6922b834c79c62b61'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:07.069974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f68e3905a82644e6922b834c79c62b61'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:07.070960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f68e3905a82644e6922b834c79c62b61'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:07.076705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:INSERT=6,keystone:SELECT=86,keystone:DELETE=8 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:07.102113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f68e3905a82644e6922b834c79c62b61'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:07.133007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f68e3905a82644e6922b834c79c62b61'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:07.146396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 928/1825] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:06 2022] DELETE /identity/v3/users/f68e3905a82644e6922b834c79c62b61 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:07.161374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:07.161859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:07.162022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:07.196246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:07.197106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:07.316966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 898/1826] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:07.426461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:07.428127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:07.611510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb9074e-b118-434e-99c5-cf1eade4ecc4 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:07.618089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:07.619004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:07.619374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:07.619912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:07.632604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:07.636572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:07.650697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 929/1827] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:08.383256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:08.385132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:08.398931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0920763-c737-41b8-b077-29f3173fe01a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:08.405099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:08.405988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:08.406272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:08.406585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] PATH_INFO: `/v3/users/2d6f736ec2844a1db8fb1a0c85ba4766` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:08.407786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d6f736ec2844a1db8fb1a0c85ba4766)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:08.409789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:08.472882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:08.474451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:08.477676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d6f736ec2844a1db8fb1a0c85ba4766'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.487537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d6f736ec2844a1db8fb1a0c85ba4766'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.488661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96d14f5-5c90-4bdc-b091-b98537eebb8e service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:08.495593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:08.496502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:08.496793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:08.497066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] PATH_INFO: `/v3/projects/cdf933c5a44747eba680d9420d6da997` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:08.498347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] RBAC: Authorizing `identity:delete_project(project_id=cdf933c5a44747eba680d9420d6da997)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:08.498496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d6f736ec2844a1db8fb1a0c85ba4766'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.499031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d6f736ec2844a1db8fb1a0c85ba4766'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.500382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:08.516945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:08.520535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7e091000-8a92-4e1b-b4cd-5663706b4322 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d6f736ec2844a1db8fb1a0c85ba4766'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.529146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 899/1828] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:08 2022] DELETE /identity/v3/users/2d6f736ec2844a1db8fb1a0c85ba4766 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:08.534020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:08.534310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:08.534629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:08.554599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:08.556004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:08.557764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cdf933c5a44747eba680d9420d6da997', 'request_id': 'req-0b128f6d-c264-449f-ba44-3440d0e0181f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.567323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cdf933c5a44747eba680d9420d6da997', 'request_id': 'req-0b128f6d-c264-449f-ba44-3440d0e0181f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.573940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cdf933c5a44747eba680d9420d6da997', 'request_id': 'req-0b128f6d-c264-449f-ba44-3440d0e0181f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.586387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] The token cache is being invalidate because project cdf933c5a44747eba680d9420d6da997 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:08.586691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.587355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.589128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 930/1829] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:08 2022] DELETE /identity/v3/projects/cdf933c5a44747eba680d9420d6da997 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:08.593446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:08.594800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:08.622386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 900/1830] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:08.632068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:08.633711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:08.696600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b128f6d-c264-449f-ba44-3440d0e0181f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:08.703197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:08.704014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:08.706092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:08.706092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] PATH_INFO: `/v3/users/b50250ffd62a4addb60c4e56a9e59fef` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:08.706879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b50250ffd62a4addb60c4e56a9e59fef)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:08.709248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:08.712601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:08.712764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919a9368-4521-4d02-9aae-d8b45d3ba4b8 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:08.721039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:08.721920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:08.722540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:08.723042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:08.747269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:08.749765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:08.763160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 901/1831] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:08.826963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b50250ffd62a4addb60c4e56a9e59fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.846310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b50250ffd62a4addb60c4e56a9e59fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.847232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b50250ffd62a4addb60c4e56a9e59fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.869204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b50250ffd62a4addb60c4e56a9e59fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.889091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b50250ffd62a4addb60c4e56a9e59fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:08.897182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 931/1832] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:08 2022] DELETE /identity/v3/users/b50250ffd62a4addb60c4e56a9e59fef => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:09.424696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:09.426421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:09.442406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce35b19-4470-4231-98b0-1130d311cfec service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:09.564379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:09.565526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:09.566036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:09.566606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] PATH_INFO: `/v3/projects/cd47264e0cf94160ae4f6330bde0ff87` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:09.568690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd47264e0cf94160ae4f6330bde0ff87)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:09.571538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:09.665284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=4,keystone:SELECT=96 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:09.667347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd47264e0cf94160ae4f6330bde0ff87', 'request_id': 'req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.676928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd47264e0cf94160ae4f6330bde0ff87', 'request_id': 'req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.687525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd47264e0cf94160ae4f6330bde0ff87', 'request_id': 'req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.708229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] The token cache is being invalidate because project cd47264e0cf94160ae4f6330bde0ff87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:09.708773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.709761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.712230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 902/1833] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:09 2022] DELETE /identity/v3/projects/cd47264e0cf94160ae4f6330bde0ff87 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:09.716719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:09.718498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:09.754602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a47a6d-ae5a-4f6f-8a7e-4111c7528e31 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:09.774997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:09.777317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:09.880210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:09.881551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:09.881927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:09.882211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] PATH_INFO: `/v3/users/9be0c39097c44929ac97e77d906d9d4c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:09.884418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9be0c39097c44929ac97e77d906d9d4c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:09.887014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:09.913340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6c577f-6a36-4c6f-aa51-071f5df0b8ca admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:09.919009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:09.919976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:09.920367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:09.920980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] PATH_INFO: `/v3/projects/7916069778ba4780853c737fe4c61c75` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:09.922418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7916069778ba4780853c737fe4c61c75)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:09.924576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:09.954480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9be0c39097c44929ac97e77d906d9d4c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.962414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9be0c39097c44929ac97e77d906d9d4c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.963877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9be0c39097c44929ac97e77d906d9d4c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.971709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9be0c39097c44929ac97e77d906d9d4c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.984887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b44d755e-5b14-44bb-827c-271f932387c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9be0c39097c44929ac97e77d906d9d4c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.986621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:09.987390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7916069778ba4780853c737fe4c61c75', 'request_id': 'req-d75369a9-d062-457d-9282-687c7ebb3600'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:09.993102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 932/1834] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:09 2022] DELETE /identity/v3/users/9be0c39097c44929ac97e77d906d9d4c => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:09.994059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7916069778ba4780853c737fe4c61c75', 'request_id': 'req-d75369a9-d062-457d-9282-687c7ebb3600'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.008801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7916069778ba4780853c737fe4c61c75', 'request_id': 'req-d75369a9-d062-457d-9282-687c7ebb3600'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.029023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] The token cache is being invalidate because project 7916069778ba4780853c737fe4c61c75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:10.029333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.030031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d75369a9-d062-457d-9282-687c7ebb3600 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.031708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 903/1835] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:09 2022] DELETE /identity/v3/projects/7916069778ba4780853c737fe4c61c75 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:10.037128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f53d1-5d09-4844-b9c9-e15c6edbeb3f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.037498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f53d1-5d09-4844-b9c9-e15c6edbeb3f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.037795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f53d1-5d09-4844-b9c9-e15c6edbeb3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.066788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fc8f53d1-5d09-4844-b9c9-e15c6edbeb3f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:10.069147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fc8f53d1-5d09-4844-b9c9-e15c6edbeb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:10.201997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 933/1836] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:10.217697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d999b4-b5e5-4331-aee7-50c683257fa0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.218703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d999b4-b5e5-4331-aee7-50c683257fa0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.219003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d999b4-b5e5-4331-aee7-50c683257fa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.265797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-14d999b4-b5e5-4331-aee7-50c683257fa0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:10.267356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-14d999b4-b5e5-4331-aee7-50c683257fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:10.281598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 904/1837] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:10.283768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f53d1-5d09-4844-b9c9-e15c6edbeb3f None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.299598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f53d1-5d09-4844-b9c9-e15c6edbeb3f None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:10.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:10.308618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.308944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.309210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.310019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:10.310333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:10.312560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:10.330481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 934/1838] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:10.333562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d999b4-b5e5-4331-aee7-50c683257fa0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.344446 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d999b4-b5e5-4331-aee7-50c683257fa0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:10.350544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:10.351411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.352072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.352348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.353233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:10.356295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:10.386734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.389440 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:10.403139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 905/1839] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:10 2022] POST /identity/v3/projects => generated 381 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:10.404857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.409431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4fa69a-5ef8-4714-801f-9c94fac73902 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:10.471866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:10.505771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:10.508025 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.508470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.508897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] PATH_INFO: `/v3/users/1eb9a16ea2e24da29c2e26a19a2e5a20` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.511102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1eb9a16ea2e24da29c2e26a19a2e5a20)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:10.514760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:10.516937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce852fa0-dd16-4fc0-a067-7fb934a5425b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:10.538523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:10.540465 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.540603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.540789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.541628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:10.549996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:10.584270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 906/1840] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:10 2022] POST /identity/v3/users => generated 291 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:10.585899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.602954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c67a40a-5b00-4bca-be48-a957d608a9a8 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:10.607962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:10.608652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.608941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.609222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.609966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:10.612071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:10.630250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 907/1841] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:10 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:10.631814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.632523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1eb9a16ea2e24da29c2e26a19a2e5a20'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.644980 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bed1ec7-6709-410f-9843-1e5028f4d8d9 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:10.653165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1eb9a16ea2e24da29c2e26a19a2e5a20'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.660731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1eb9a16ea2e24da29c2e26a19a2e5a20'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.662868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:10.663862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.664501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.665054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.666084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:10.673720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:10.674712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1eb9a16ea2e24da29c2e26a19a2e5a20'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.688955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 908/1842] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:10 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:10.690984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.693165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:10.701465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f196e43e-49e0-475a-b32c-d91af228249c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1eb9a16ea2e24da29c2e26a19a2e5a20'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:10.723667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 935/1843] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:10 2022] DELETE /identity/v3/users/1eb9a16ea2e24da29c2e26a19a2e5a20 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:10.737254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ff4609-91a4-40a3-adb2-f0360e553a44 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:10.762887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b4881-4217-4c73-8892-3de0474ded2a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.766105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b4881-4217-4c73-8892-3de0474ded2a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.767405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6b4881-4217-4c73-8892-3de0474ded2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.787493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:10.807579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ca6b4881-4217-4c73-8892-3de0474ded2a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:10.810515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ca6b4881-4217-4c73-8892-3de0474ded2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:10.882943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:10.884060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.884528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.885844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] PATH_INFO: `/v3/projects/78d4886c6e394511a1c64b59f3e5ee7a/users/7fca3991b6e3465183530233a78f3354/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:10.918887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] RBAC: Authorizing `identity:create_grant(project_id=78d4886c6e394511a1c64b59f3e5ee7a, user_id=7fca3991b6e3465183530233a78f3354, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:10.922217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:10.960371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 909/1844] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:10 2022] PUT /identity/v3/projects/78d4886c6e394511a1c64b59f3e5ee7a/users/7fca3991b6e3465183530233a78f3354/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:10.961962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:10.974811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:10.992667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 936/1845] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:10.992667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:10.992667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:10.992667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.046811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] MFA Rules not processed for user `7fca3991b6e3465183530233a78f3354`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:11.048491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:11.136450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e12d10-8f1e-490a-8c3d-6a48fc88680a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.146961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.147924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.148220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.148491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] PATH_INFO: `/v3/projects/252a54011bd742e787a34e5c4e4cb570` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.152296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] RBAC: Authorizing `identity:delete_project(project_id=252a54011bd742e787a34e5c4e4cb570)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.154535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.190020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:11.205915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 937/1846] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:10 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:11.208571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.210875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:11.238867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '252a54011bd742e787a34e5c4e4cb570', 'request_id': 'req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:11.247391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '252a54011bd742e787a34e5c4e4cb570', 'request_id': 'req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:11.257548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '252a54011bd742e787a34e5c4e4cb570', 'request_id': 'req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:11.273092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] The token cache is being invalidate because project 252a54011bd742e787a34e5c4e4cb570 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:11.274065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:11.274517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:11.277437 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 910/1847] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:10 2022] DELETE /identity/v3/projects/252a54011bd742e787a34e5c4e4cb570 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:11.278360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.282116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:11.287297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37014e9-6866-43ae-a0d2-8cabd4e4ace4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.355941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.356238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.356492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.356767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.356902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce9ea9-76c1-4965-80ba-55e3e09c6147 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.358666 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:11.361915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4718debc-b3f4-4ca8-b48a-55859693c878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.362709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4718debc-b3f4-4ca8-b48a-55859693c878 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.363071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4718debc-b3f4-4ca8-b48a-55859693c878 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.363444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4718debc-b3f4-4ca8-b48a-55859693c878 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.364275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4718debc-b3f4-4ca8-b48a-55859693c878 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.366343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4718debc-b3f4-4ca8-b48a-55859693c878 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.372594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:11.377547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.387650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.404578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 911/1848] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:11 2022] POST /identity/v3/projects => generated 383 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:11.407971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0e698a-4934-4785-9817-155b600b8b93 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.408499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0e698a-4934-4785-9817-155b600b8b93 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.409049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0e698a-4934-4785-9817-155b600b8b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.417605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 938/1849] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:11.419995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.436765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ab0e698a-4934-4785-9817-155b600b8b93 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:11.438552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ab0e698a-4934-4785-9817-155b600b8b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:11.442327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af5675-c214-4f34-b5af-ba55097cc6a6 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.520470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 912/1850] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:11.523044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ac752-2b8d-4398-8ed9-c2c9b1beb856 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.523334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ac752-2b8d-4398-8ed9-c2c9b1beb856 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.523664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ac752-2b8d-4398-8ed9-c2c9b1beb856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.547705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ca1ac752-2b8d-4398-8ed9-c2c9b1beb856 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:11.548959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ca1ac752-2b8d-4398-8ed9-c2c9b1beb856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:11.565328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 913/1851] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:11.565889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1ac752-2b8d-4398-8ed9-c2c9b1beb856 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.589625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1ac752-2b8d-4398-8ed9-c2c9b1beb856 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.592268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.592977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.593265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.593554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.594271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.599537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.601069 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.602240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.602905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.604580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:11.605861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.611909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.627077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 939/1852] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:11 2022] POST /identity/v3/users => generated 298 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:11.637056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.643453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 914/1853] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:11.644594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.678538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179b8faf-cd4a-4965-b2d0-f4491035513e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.679732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defc2f7c-83ab-4aa5-8bd9-3900e12296ad admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.690954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.692427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.692660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.693317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.694092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.695211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.696096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.696716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.697274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.699847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.700734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.706056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.735135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 940/1854] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:11 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:11.738225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.743658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=7,keystone:SELECT=91,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:11.764854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356b021-d9d8-42fd-b4d0-1bd0f9a6c23c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:11.774570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 915/1855] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:11 2022] POST /identity/v3/projects => generated 379 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:11.778696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:11.779719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.780907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:11.781458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:11.782059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] PATH_INFO: `/v3/projects/fc4faa647dc54781aee53e4efbf61792/users/2f7c71e8bd474d43b02a2e61122ba285/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:11.818294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc4faa647dc54781aee53e4efbf61792, user_id=2f7c71e8bd474d43b02a2e61122ba285, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:11.825625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:11.860363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 941/1856] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:11 2022] PUT /identity/v3/projects/fc4faa647dc54781aee53e4efbf61792/users/2f7c71e8bd474d43b02a2e61122ba285/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:11.862614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:11.884859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d077dee-92ac-454c-878d-0e99b618314b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d262909-055f-49db-b661-c99217138c96 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.040560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.041446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.041783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.042119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.042933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.057843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.058320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.059064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.059347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.059612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.060320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.062971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.073910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 942/1857] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:11 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:12.078711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:12.084305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:12.118636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37aa6e28-41bc-4bee-afb3-24c59b2a6a08 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.123862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 916/1858] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:11 2022] POST /identity/v3/users => generated 290 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:12.124781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:12.127963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.129034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.129566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.130294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] PATH_INFO: `/v3/projects/28ebea9991bb48eda27cde2776d05a22` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.131710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28ebea9991bb48eda27cde2776d05a22)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.138651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.163214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f011ff2e-59df-4eb9-944f-f9eab4a42e63 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.184313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.186075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.187946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.188550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] PATH_INFO: `/v3/projects/fc4faa647dc54781aee53e4efbf61792/users/2f7c71e8bd474d43b02a2e61122ba285/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.192883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc4faa647dc54781aee53e4efbf61792, user_id=2f7c71e8bd474d43b02a2e61122ba285, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.199217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.201256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:12.215980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28ebea9991bb48eda27cde2776d05a22', 'request_id': 'req-4e964be5-1775-4dca-ac20-5568e8269af0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.230800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28ebea9991bb48eda27cde2776d05a22', 'request_id': 'req-4e964be5-1775-4dca-ac20-5568e8269af0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.240112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28ebea9991bb48eda27cde2776d05a22', 'request_id': 'req-4e964be5-1775-4dca-ac20-5568e8269af0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.258526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] The token cache is being invalidate because project 28ebea9991bb48eda27cde2776d05a22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:12.258857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.259687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.262308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 943/1859] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:12 2022] DELETE /identity/v3/projects/28ebea9991bb48eda27cde2776d05a22 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:12.263147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:12.265971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:12.287009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 917/1860] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:12 2022] PUT /identity/v3/projects/fc4faa647dc54781aee53e4efbf61792/users/2f7c71e8bd474d43b02a2e61122ba285/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:12.288734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:12.303795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:12.359121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e964be5-1775-4dca-ac20-5568e8269af0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.467338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b0f25a-3102-4558-b60a-84dd3877e886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.467338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b0f25a-3102-4558-b60a-84dd3877e886 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.467338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b0f25a-3102-4558-b60a-84dd3877e886 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.467338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b0f25a-3102-4558-b60a-84dd3877e886 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.467957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b0f25a-3102-4558-b60a-84dd3877e886 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.467957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b0f25a-3102-4558-b60a-84dd3877e886 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.499034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 944/1861] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:12.499034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d55c3e-0d23-4028-b3de-2f7ce0e360e3 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.499034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d55c3e-0d23-4028-b3de-2f7ce0e360e3 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.499034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d55c3e-0d23-4028-b3de-2f7ce0e360e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.499391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b625d64c-4b78-42eb-895c-0885e9e20969 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.513891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.513891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.513891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.514293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] PATH_INFO: `/v3/users/8c071c176fe646f692404ba9956d596b` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.515543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c071c176fe646f692404ba9956d596b)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.517519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.572550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e3d55c3e-0d23-4028-b3de-2f7ce0e360e3 None None] MFA Rules not processed for user `2f7c71e8bd474d43b02a2e61122ba285`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:12.575236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e3d55c3e-0d23-4028-b3de-2f7ce0e360e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:12.586137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:INSERT=5,keystone:SELECT=93,keystone:DELETE=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:12.628223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c071c176fe646f692404ba9956d596b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.636005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c071c176fe646f692404ba9956d596b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.645139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c071c176fe646f692404ba9956d596b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.645970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c071c176fe646f692404ba9956d596b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.665931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:12.685601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 945/1862] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:12 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:12.687315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d55c3e-0d23-4028-b3de-2f7ce0e360e3 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:12.697629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c071c176fe646f692404ba9956d596b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:12.700352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d55c3e-0d23-4028-b3de-2f7ce0e360e3 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.705971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.706723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 918/1863] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:12 2022] DELETE /identity/v3/users/8c071c176fe646f692404ba9956d596b => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:12.706864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.707162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.707472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.707552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:12.708631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.708895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:12.711970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.727405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 946/1864] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:12.729373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:12.731912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:12.749048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a8675a-fa93-4a75-a6a1-dcc213d6d762 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.788268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2d98c3-81c1-4676-bccb-77f94ad608a5 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:12.793076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.793868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.794262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.794665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.796625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:12.820371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1044c94-988f-4c5c-bf30-d93d1b178369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:12.821591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1044c94-988f-4c5c-bf30-d93d1b178369 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.821879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1044c94-988f-4c5c-bf30-d93d1b178369 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.822162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1044c94-988f-4c5c-bf30-d93d1b178369 admin admin] PATH_INFO: `/v3/projects/ea4bc771273b46aa8dfda3b4c5f35c42/users/82bf4af5ee04409ebc9aff1ec01a384d/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.839844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1044c94-988f-4c5c-bf30-d93d1b178369 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea4bc771273b46aa8dfda3b4c5f35c42, user_id=82bf4af5ee04409ebc9aff1ec01a384d, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.842592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1044c94-988f-4c5c-bf30-d93d1b178369 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.860686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 947/1865] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:12 2022] PUT /identity/v3/projects/ea4bc771273b46aa8dfda3b4c5f35c42/users/82bf4af5ee04409ebc9aff1ec01a384d/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:12.871366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa4473-799a-490d-9ad3-ff18ab03bb24 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:12.871758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa4473-799a-490d-9ad3-ff18ab03bb24 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:12.872197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa4473-799a-490d-9ad3-ff18ab03bb24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:12.889890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:12.891575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:12.906023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 919/1866] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:12 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:12.922230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9daa4473-799a-490d-9ad3-ff18ab03bb24 None None] MFA Rules not processed for user `82bf4af5ee04409ebc9aff1ec01a384d`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:12.923546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9daa4473-799a-490d-9ad3-ff18ab03bb24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:13.032918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 948/1867] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:12 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:13.034241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.042820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa4473-799a-490d-9ad3-ff18ab03bb24 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.048527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be32742-d5cf-40ed-b730-407265babf83 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.124151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:13.144978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.145740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.146046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.146309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.147254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.148792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa4473-799a-490d-9ad3-ff18ab03bb24 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.150525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.153429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.154797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.154940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.155300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.157281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:13.170857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:13.187358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 920/1868] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:13 2022] POST /identity/v3/projects => generated 379 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:13.189373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.246237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.248390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.279081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63670f76-7a2f-4ac3-96d7-7544548b0b5f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.279680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 949/1869] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:13 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:13.286461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.286979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.287379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.287855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.288643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.292352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.319377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 921/1870] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:13 2022] POST /identity/v3/users => generated 290 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:13.324519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.346471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a884a0-c822-4728-b268-04d0f7f52cc7 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.352993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.354593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.354971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.355328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.357041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.358856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.377188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 950/1871] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:13.381425 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.393463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9affb5b-af74-4371-8cfc-94ac3131c8eb admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.403119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.404055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.404580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.404881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.406259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.411630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.415744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:13.432778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.475885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac3db1e-46db-49da-9db2-6bcbe2ab521b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.492621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 922/1872] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:13.494121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.501611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.502953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.503661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.504227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] PATH_INFO: `/v3/projects/5c69e33d94b04656908a7041cc37f11e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.506791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c69e33d94b04656908a7041cc37f11e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.511162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.518337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4567fda-5f24-4b8a-81fc-76c691923a77 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.530312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a6b8ec-6112-4871-8130-264715e12946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.531267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a6b8ec-6112-4871-8130-264715e12946 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.531557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a6b8ec-6112-4871-8130-264715e12946 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.531891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a6b8ec-6112-4871-8130-264715e12946 admin admin] PATH_INFO: `/v3/projects/3004d4d809844046add9180ebf2d7eda/users/776f6073f0ea44bcb8024f6fe0d0d55b/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.570461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a6b8ec-6112-4871-8130-264715e12946 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3004d4d809844046add9180ebf2d7eda, user_id=776f6073f0ea44bcb8024f6fe0d0d55b, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.575460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a6b8ec-6112-4871-8130-264715e12946 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.589603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c69e33d94b04656908a7041cc37f11e', 'request_id': 'req-165f9aab-01b1-4b21-b546-6bc98af19d06'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:13.600262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c69e33d94b04656908a7041cc37f11e', 'request_id': 'req-165f9aab-01b1-4b21-b546-6bc98af19d06'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:13.605778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 923/1873] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:13 2022] PUT /identity/v3/projects/3004d4d809844046add9180ebf2d7eda/users/776f6073f0ea44bcb8024f6fe0d0d55b/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:13.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c69e33d94b04656908a7041cc37f11e', 'request_id': 'req-165f9aab-01b1-4b21-b546-6bc98af19d06'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:13.620210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.620633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.621172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.626710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] The token cache is being invalidate because project 5c69e33d94b04656908a7041cc37f11e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:13.627231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:13.628040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-165f9aab-01b1-4b21-b546-6bc98af19d06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:13.631119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 951/1874] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:13 2022] DELETE /identity/v3/projects/5c69e33d94b04656908a7041cc37f11e => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:13.636738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c6b3e5-2680-474a-854e-5e7424590c3e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.637281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c6b3e5-2680-474a-854e-5e7424590c3e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.637700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c6b3e5-2680-474a-854e-5e7424590c3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.667663 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-08c6b3e5-2680-474a-854e-5e7424590c3e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:13.672070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-08c6b3e5-2680-474a-854e-5e7424590c3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:13.724104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] MFA Rules not processed for user `776f6073f0ea44bcb8024f6fe0d0d55b`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:13.726257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:13.840145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 952/1875] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:13.848508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bf273-adaa-47d6-b330-5b6a1232ce35 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.849011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bf273-adaa-47d6-b330-5b6a1232ce35 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.849401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bf273-adaa-47d6-b330-5b6a1232ce35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.869921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-735bf273-adaa-47d6-b330-5b6a1232ce35 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:13.871300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-735bf273-adaa-47d6-b330-5b6a1232ce35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:13.900085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 953/1876] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:13.905015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735bf273-adaa-47d6-b330-5b6a1232ce35 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.912384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 924/1877] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:13 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:13.913651 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:13.915207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:13.927236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735bf273-adaa-47d6-b330-5b6a1232ce35 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.946886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.948484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.948877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.949291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.950931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:13.955666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbf1bbf-01ab-4758-85f1-10877a5fa6c0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:13.956030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.958092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:13.976399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:13.977095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:13.977389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:13.977675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:13.978890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:13.980640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.001613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:14.018086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 954/1878] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:14.019408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.032729 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 925/1879] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:13 2022] POST /identity/v3/projects => generated 377 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:14.032986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:14.033728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.037853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c6b85-bca8-4a4c-80b3-afb025d535d5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.168722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.169467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.169762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.170044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.173640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.178716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.186502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05669dea-9f07-4242-be96-0f97d87e8c1d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.201773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.202632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.202926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.204378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.207873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.213347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.238231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 955/1880] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:14 2022] POST /identity/v3/projects => generated 371 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:14.242469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.269433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 926/1881] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:14 2022] POST /identity/v3/users => generated 295 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:14.270810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.284075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f2f311-ef6f-4c06-b119-854c565c4a5b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.395992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.397396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.397848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.398009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610b35c0-0a99-4740-844c-a64a78ee152c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.399777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.402988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.408291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.408721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.409763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.410144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.410463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.412167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.424371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.425710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 927/1882] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:14 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:14.432272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.456399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38e7d4b-c372-4ceb-81cb-413398897fd2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.467501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 956/1883] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:14 2022] POST /identity/v3/users => generated 286 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:14.468614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.472520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.473442 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.473734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.474151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] PATH_INFO: `/v3/projects/bd97a99ee0da4c71bfd57552417d956f/users/471305aa418a451199ce21b8a201b217/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.480403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab781b3-fccc-445a-9ccb-07c0588052c8 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.485452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.486251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.486714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.487347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.488290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.490048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.503452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd97a99ee0da4c71bfd57552417d956f, user_id=471305aa418a451199ce21b8a201b217, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.507690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 957/1884] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:14.510893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.511023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.530747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a30fc50-f3db-47d2-896b-75398403259d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.537680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 928/1885] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:14 2022] PUT /identity/v3/projects/bd97a99ee0da4c71bfd57552417d956f/users/471305aa418a451199ce21b8a201b217/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:14.543005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.628340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.629678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.630408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.630977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.632212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.634029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.637523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:14.647170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4903465-1d53-4cf7-8c53-b447e6b10dd8 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.653410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 958/1886] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:14 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:14.655461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.658645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.659897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.660331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.660746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.664151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:14.666716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.667586 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6c7f47-313f-463e-9e67-f8879e51a5a3 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.671207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.672860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57822ee3-3c5f-444b-aafa-af9ff13d7c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.673783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57822ee3-3c5f-444b-aafa-af9ff13d7c9e admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.674073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57822ee3-3c5f-444b-aafa-af9ff13d7c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.674357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57822ee3-3c5f-444b-aafa-af9ff13d7c9e admin admin] PATH_INFO: `/v3/projects/832778af1b9a431fa277ed974f10ecfd/users/a1a614eee78e4796ba1a5aa52d43f62a/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.695258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 929/1887] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:14 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:14.696528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57822ee3-3c5f-444b-aafa-af9ff13d7c9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=832778af1b9a431fa277ed974f10ecfd, user_id=a1a614eee78e4796ba1a5aa52d43f62a, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.700013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57822ee3-3c5f-444b-aafa-af9ff13d7c9e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.700805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.715329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319a0a61-d1be-45a5-b9f8-bedd2c526795 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:14.716566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 959/1888] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:14 2022] PUT /identity/v3/projects/832778af1b9a431fa277ed974f10ecfd/users/a1a614eee78e4796ba1a5aa52d43f62a/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:14.728086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.728470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.729661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.766512 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] MFA Rules not processed for user `a1a614eee78e4796ba1a5aa52d43f62a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:14.767874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:14.799204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8814b456-9a5a-432a-ba82-f17c459ec402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:14.800721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8814b456-9a5a-432a-ba82-f17c459ec402 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.801683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8814b456-9a5a-432a-ba82-f17c459ec402 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.802380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8814b456-9a5a-432a-ba82-f17c459ec402 admin admin] PATH_INFO: `/v3/projects/bd97a99ee0da4c71bfd57552417d956f/users/471305aa418a451199ce21b8a201b217/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.804859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8814b456-9a5a-432a-ba82-f17c459ec402 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd97a99ee0da4c71bfd57552417d956f, user_id=471305aa418a451199ce21b8a201b217, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:14.808346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8814b456-9a5a-432a-ba82-f17c459ec402 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:14.822689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 930/1889] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:14 2022] PUT /identity/v3/projects/bd97a99ee0da4c71bfd57552417d956f/users/471305aa418a451199ce21b8a201b217/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:14.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:14.835690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:14.836161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:14.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 960/1890] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:14 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:14.872207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:14.873745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:14.876240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] MFA Rules not processed for user `471305aa418a451199ce21b8a201b217`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:14.877807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:14.946011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e49f52-4312-4c46-a812-02c70d12bb1d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:15.010562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:15.011241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:15.011531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:15.012019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:15.019579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 931/1891] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:14 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:15.118722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:15.121169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:15.139719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ad19c4-2d44-490e-b0bf-e1bb492b2497 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:15.144914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:15.145999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:15.146541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:15.147056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:15.147602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:15.172837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:15.177131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:15.195615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:15.199371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:15.208715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 932/1892] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:15 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:15.236648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 961/1893] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:14 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:15.261152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:15.289935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:15.302042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf20939c-5a0f-4a61-9200-4c9774614964 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:15.315302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ea8816-6e97-4408-8457-f6c419aced4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:15.317019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ea8816-6e97-4408-8457-f6c419aced4f service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:15.318646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ea8816-6e97-4408-8457-f6c419aced4f service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:15.319196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ea8816-6e97-4408-8457-f6c419aced4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:15.323817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-55ea8816-6e97-4408-8457-f6c419aced4f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:15.363428 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:15.387998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7943e648-ad4a-48ec-9a6f-7ebce8509e97 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:15.404107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:15.405418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:15.406040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:15.406397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] PATH_INFO: `/v3/users/a1a614eee78e4796ba1a5aa52d43f62a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:15.408353 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1a614eee78e4796ba1a5aa52d43f62a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:15.412412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:15.503313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1a614eee78e4796ba1a5aa52d43f62a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:15.521814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ea8816-6e97-4408-8457-f6c419aced4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:15.522264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1a614eee78e4796ba1a5aa52d43f62a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:15.523294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ea8816-6e97-4408-8457-f6c419aced4f service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:15.546935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1a614eee78e4796ba1a5aa52d43f62a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:15.547741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1a614eee78e4796ba1a5aa52d43f62a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:15.549754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 962/1894] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:15 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:15.588722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1a614eee78e4796ba1a5aa52d43f62a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:15.599885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 933/1895] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:15 2022] DELETE /identity/v3/users/a1a614eee78e4796ba1a5aa52d43f62a => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:15.601632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:15.601927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:15.602212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:15.646481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:15.647883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:15.836688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 963/1896] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:15.856832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:15.858377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:15.951218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3e30c-6961-4da9-b7eb-5509cc1b82b7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:15.957317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:15.959511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:15.959823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:15.960133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:15.975877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:15.978183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:15.993913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 934/1897] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:16.822480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:16.826261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:16.847481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:16.849232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:16.855789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc1499d-4660-44d2-ac36-e7b43ee222ee None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:16.870232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133f1c30-29bd-4d06-95fc-a889d52f82c5 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:16.879591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:16.881610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:16.881610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:16.881807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] PATH_INFO: `/v3/projects/832778af1b9a431fa277ed974f10ecfd` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:16.887582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=832778af1b9a431fa277ed974f10ecfd)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:16.890819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:16.898325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:16.900197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:16.900748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:16.901195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] PATH_INFO: `/v3/users/7fca3991b6e3465183530233a78f3354` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:16.903316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7fca3991b6e3465183530233a78f3354)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:16.905917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:16.982219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '832778af1b9a431fa277ed974f10ecfd', 'request_id': 'req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:16.991441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7fca3991b6e3465183530233a78f3354'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:16.994996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '832778af1b9a431fa277ed974f10ecfd', 'request_id': 'req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.003312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7fca3991b6e3465183530233a78f3354'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.004240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '832778af1b9a431fa277ed974f10ecfd', 'request_id': 'req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.017046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7fca3991b6e3465183530233a78f3354'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.018082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7fca3991b6e3465183530233a78f3354'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.024430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] The token cache is being invalidate because project 832778af1b9a431fa277ed974f10ecfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:17.025095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.026989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ca3e3fb1-d2d2-417c-8709-8531c4147ee2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.029265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 964/1898] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:16 2022] DELETE /identity/v3/projects/832778af1b9a431fa277ed974f10ecfd => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:17.034935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f77764-a1ec-41eb-b463-c0d0f55bd835 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.035515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f77764-a1ec-41eb-b463-c0d0f55bd835 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.035662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:17.036124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f77764-a1ec-41eb-b463-c0d0f55bd835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.043217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-95a22ddd-a5b9-469e-ad98-452636153cc2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7fca3991b6e3465183530233a78f3354'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:17.044612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:17.055044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 935/1899] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:16 2022] DELETE /identity/v3/users/7fca3991b6e3465183530233a78f3354 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:17.060280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.060637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.060942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.064876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e5f77764-a1ec-41eb-b463-c0d0f55bd835 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:17.066298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e5f77764-a1ec-41eb-b463-c0d0f55bd835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:17.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:17.089597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:17.177648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 965/1900] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:17.179894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e070e2-2607-4aa1-a491-a75099e0ab40 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.181008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e070e2-2607-4aa1-a491-a75099e0ab40 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.181499 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e070e2-2607-4aa1-a491-a75099e0ab40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.206698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 936/1901] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:17.214403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-63e070e2-2607-4aa1-a491-a75099e0ab40 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:17.215730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-63e070e2-2607-4aa1-a491-a75099e0ab40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:17.222716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.224155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:17.232622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 966/1902] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:17.236236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e070e2-2607-4aa1-a491-a75099e0ab40 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.249629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e070e2-2607-4aa1-a491-a75099e0ab40 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:17.272351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:17.273019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.273311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.273589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.274345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:17.274732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:17.276922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:17.296075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 967/1903] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:41:17.297498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.309467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc86141-644a-4646-8c78-e974dfb73d7f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:17.331715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:17.331715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.331715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.331715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.332201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:17.332201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:17.337815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ef0e2e-3119-4930-b9f6-44ccd14590e7 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:17.344096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:17.344960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.345088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.345419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.368510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:17.370927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:17.392010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 968/1904] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:17 2022] POST /identity/v3/projects => generated 369 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:17.393341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.395977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 937/1905] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:17.500658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc8444e-5909-4f60-bea4-4ece826aeecb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:17.506972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:17.507667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.507955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.508237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.509122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:17.511248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:17.560495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 969/1906] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:17 2022] POST /identity/v3/users => generated 285 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:17.561726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.573904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a37145-6555-4414-88d3-80dcc164681f service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:17.579443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:17.580461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.580809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.581157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.582012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:17.583837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:17.610590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 938/1907] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:17 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:17.611958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.624854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df2205-e242-46a6-beb8-3285c40c1137 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:17.638799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056bdbb4-9ede-4c41-8724-fa4e8b51cba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:17.639502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bdbb4-9ede-4c41-8724-fa4e8b51cba2 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.639794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bdbb4-9ede-4c41-8724-fa4e8b51cba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.640266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bdbb4-9ede-4c41-8724-fa4e8b51cba2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.640809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056bdbb4-9ede-4c41-8724-fa4e8b51cba2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:17.642459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056bdbb4-9ede-4c41-8724-fa4e8b51cba2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:17.669823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 970/1908] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:17 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:17.671229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.684724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14be645-ce4a-4be6-8fd7-8f58e4472bd4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:17.689913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:17.690472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.690763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.691046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] PATH_INFO: `/v3/projects/52bd3580c4c342ebae801d2b710ac508/users/6736db4b6ba0481c921e1107dd87faf6/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.714482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52bd3580c4c342ebae801d2b710ac508, user_id=6736db4b6ba0481c921e1107dd87faf6, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:17.716360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:17.734302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 939/1909] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:17 2022] PUT /identity/v3/projects/52bd3580c4c342ebae801d2b710ac508/users/6736db4b6ba0481c921e1107dd87faf6/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:17.746299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:17.747241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:17.747776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:17.830492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] MFA Rules not processed for user `6736db4b6ba0481c921e1107dd87faf6`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:17.833180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:17.912055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:17.982773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 971/1910] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:17 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:17.996019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:17.998024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:18.090559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:18.110589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8f6c7e-50ba-4429-9114-0d4f6a82f911 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:18.117885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fb6c2e2ddb3b4d488fa25fa8e258fec0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:18.118836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:18.119306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:18.119844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:18.131567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:18.134085 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:18.145640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 940/1911] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:17 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:18.339574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:18.342416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:18.501208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919f03ee-6db8-43f5-9f34-e6b7f6de63c2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:18.518829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:18.519735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:18.520065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:18.520299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] PATH_INFO: `/v3/projects/78d4886c6e394511a1c64b59f3e5ee7a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:18.521852 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78d4886c6e394511a1c64b59f3e5ee7a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:18.525069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:18.597483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78d4886c6e394511a1c64b59f3e5ee7a', 'request_id': 'req-f386504e-a938-43b5-890b-a59dceebbc39'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.630754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78d4886c6e394511a1c64b59f3e5ee7a', 'request_id': 'req-f386504e-a938-43b5-890b-a59dceebbc39'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78d4886c6e394511a1c64b59f3e5ee7a', 'request_id': 'req-f386504e-a938-43b5-890b-a59dceebbc39'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.663594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] The token cache is being invalidate because project 78d4886c6e394511a1c64b59f3e5ee7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:18.663916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.665589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.670467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 972/1912] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:18 2022] DELETE /identity/v3/projects/78d4886c6e394511a1c64b59f3e5ee7a => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:18.674759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:18.688653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a508c2-6720-45a3-8921-3f3fa75a0ce9 service glance] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:18.770034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:18.771040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:18.771347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:18.771618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] PATH_INFO: `/v3/users/2f7c71e8bd474d43b02a2e61122ba285` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:18.773775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f7c71e8bd474d43b02a2e61122ba285)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:18.777648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:18.912554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f7c71e8bd474d43b02a2e61122ba285'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.929963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f7c71e8bd474d43b02a2e61122ba285'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.948266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f7c71e8bd474d43b02a2e61122ba285'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.949585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f7c71e8bd474d43b02a2e61122ba285'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.982314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-85cc3c25-ad5f-4be1-9aef-60d7216122bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f7c71e8bd474d43b02a2e61122ba285'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:18.995594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 941/1913] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:18 2022] DELETE /identity/v3/users/2f7c71e8bd474d43b02a2e61122ba285 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:19.119417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:19.122300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:19.155732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f386504e-a938-43b5-890b-a59dceebbc39 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:19.269926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:19.273995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:19.274389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:19.275084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] PATH_INFO: `/v3/users/82bf4af5ee04409ebc9aff1ec01a384d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:19.276760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] RBAC: Authorizing `identity:delete_user(user_id=82bf4af5ee04409ebc9aff1ec01a384d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:19.279076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:19.281069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:19.401853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82bf4af5ee04409ebc9aff1ec01a384d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.419575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82bf4af5ee04409ebc9aff1ec01a384d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.420624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82bf4af5ee04409ebc9aff1ec01a384d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.433672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82bf4af5ee04409ebc9aff1ec01a384d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.450848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82bf4af5ee04409ebc9aff1ec01a384d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.468459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 973/1914] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:19 2022] DELETE /identity/v3/users/82bf4af5ee04409ebc9aff1ec01a384d => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:19.470189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:19.471131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:19.471756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:19.500220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:19.502001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:19.598037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:19.599558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:19.643321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 942/1915] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:19.650116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:19.653070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:19.676352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:19.738459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df0b2b1-e9b4-45ba-89e1-f0a5d132052d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:19.743720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:19.744985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:19.745271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:19.745553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] PATH_INFO: `/v3/projects/fc4faa647dc54781aee53e4efbf61792` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:19.747258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc4faa647dc54781aee53e4efbf61792)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:19.753828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:19.760177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d370bd-5a4a-4900-b489-5724f806f20a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:19.770809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2a8df5-9b02-4cb9-b622-354e1bfe9587 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:19.771509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2a8df5-9b02-4cb9-b622-354e1bfe9587 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:19.771873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2a8df5-9b02-4cb9-b622-354e1bfe9587 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:19.772152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2a8df5-9b02-4cb9-b622-354e1bfe9587 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:19.784684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2a8df5-9b02-4cb9-b622-354e1bfe9587 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:19.786167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2a8df5-9b02-4cb9-b622-354e1bfe9587 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:19.797152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 943/1916] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:19.833721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc4faa647dc54781aee53e4efbf61792', 'request_id': 'req-f5d8a65f-d468-480c-b442-446b48b4b711'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.846808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc4faa647dc54781aee53e4efbf61792', 'request_id': 'req-f5d8a65f-d468-480c-b442-446b48b4b711'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.853861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc4faa647dc54781aee53e4efbf61792', 'request_id': 'req-f5d8a65f-d468-480c-b442-446b48b4b711'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.868911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] The token cache is being invalidate because project fc4faa647dc54781aee53e4efbf61792 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:19.869265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.871098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f5d8a65f-d468-480c-b442-446b48b4b711 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:19.873947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 974/1917] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:19 2022] DELETE /identity/v3/projects/fc4faa647dc54781aee53e4efbf61792 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:19.879642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3912fb-ec15-403d-83b0-b172f1ca9841 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:19.879964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3912fb-ec15-403d-83b0-b172f1ca9841 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:19.880205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3912fb-ec15-403d-83b0-b172f1ca9841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:19.903205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-df3912fb-ec15-403d-83b0-b172f1ca9841 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:19.907048 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-df3912fb-ec15-403d-83b0-b172f1ca9841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.016433 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 944/1918] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:20.025971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fa1e-8ebb-48b5-89dd-fd19a7b16eb5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.026273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fa1e-8ebb-48b5-89dd-fd19a7b16eb5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.026593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fa1e-8ebb-48b5-89dd-fd19a7b16eb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.052006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4dd2fa1e-8ebb-48b5-89dd-fd19a7b16eb5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:20.053620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4dd2fa1e-8ebb-48b5-89dd-fd19a7b16eb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.067979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 975/1919] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:20.072735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3912fb-ec15-403d-83b0-b172f1ca9841 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.084485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3912fb-ec15-403d-83b0-b172f1ca9841 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.090818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.091512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.091801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.092086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.092842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:20.093157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.094931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.116197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 945/1920] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:20.117804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd2fa1e-8ebb-48b5-89dd-fd19a7b16eb5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.123638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:20.129819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd2fa1e-8ebb-48b5-89dd-fd19a7b16eb5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.138156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.138944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.139657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.139657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.142193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.142334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.161365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.164582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.181051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028ebbb-5c19-42a2-96b0-f80180883ae9 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.187325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.196141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.196546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.196905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] PATH_INFO: `/v3/projects/ea4bc771273b46aa8dfda3b4c5f35c42` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.198983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea4bc771273b46aa8dfda3b4c5f35c42)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.200111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 976/1921] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:20 2022] POST /identity/v3/projects => generated 387 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:20.201081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.201194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.274793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea4bc771273b46aa8dfda3b4c5f35c42', 'request_id': 'req-a7cdb7dd-c46f-469b-b274-b738a486203b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.280703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172c3cfd-6a0b-49b8-88bd-7e45250045c0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.282935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea4bc771273b46aa8dfda3b4c5f35c42', 'request_id': 'req-a7cdb7dd-c46f-469b-b274-b738a486203b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.293194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea4bc771273b46aa8dfda3b4c5f35c42', 'request_id': 'req-a7cdb7dd-c46f-469b-b274-b738a486203b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.306302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] The token cache is being invalidate because project ea4bc771273b46aa8dfda3b4c5f35c42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:20.306664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.307671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.312234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 946/1922] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:20 2022] DELETE /identity/v3/projects/ea4bc771273b46aa8dfda3b4c5f35c42 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:20.313271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.314824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.335242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:20.353536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.354866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.355163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.355445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.356298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.358298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.381877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 977/1923] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:20 2022] POST /identity/v3/users => generated 294 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:20.386042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.387878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.392391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdb7dd-c46f-469b-b274-b738a486203b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.397679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.398831 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.399116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.399342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] PATH_INFO: `/v3/users/776f6073f0ea44bcb8024f6fe0d0d55b` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.401262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] RBAC: Authorizing `identity:delete_user(user_id=776f6073f0ea44bcb8024f6fe0d0d55b)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.403426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.403700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7effed32-e505-4de6-af20-b0691208e100 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.409354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.410034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.410371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.410686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.411431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.413028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.426268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 978/1924] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:20 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:20.430389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.442011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d434fb35-47f7-4a12-8c39-0f9fd3ac8924 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.446925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.447947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.448338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.448715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.449553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.451276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.457628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '776f6073f0ea44bcb8024f6fe0d0d55b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.463792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 979/1925] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:20.467974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.469368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '776f6073f0ea44bcb8024f6fe0d0d55b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.482527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '776f6073f0ea44bcb8024f6fe0d0d55b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.484530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '776f6073f0ea44bcb8024f6fe0d0d55b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.496073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583a0e94-6b66-47f4-9d9f-8d7a4d4c1d49 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.502534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-534b46b9-c815-4511-a70e-6fb75341a47b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '776f6073f0ea44bcb8024f6fe0d0d55b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.502752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.503527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.503811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.504095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] PATH_INFO: `/v3/projects/d96988388648448abbf810754470bda9/users/3263800cc0234d59a259b690dc68a323/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.512547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 947/1926] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:20 2022] DELETE /identity/v3/users/776f6073f0ea44bcb8024f6fe0d0d55b => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:20.527394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d96988388648448abbf810754470bda9, user_id=3263800cc0234d59a259b690dc68a323, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.530442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.548346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 980/1927] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:20 2022] PUT /identity/v3/projects/d96988388648448abbf810754470bda9/users/3263800cc0234d59a259b690dc68a323/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:20.560466 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c785c-e273-485e-bc46-85ff5ac4d586 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.561125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c785c-e273-485e-bc46-85ff5ac4d586 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.561377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c785c-e273-485e-bc46-85ff5ac4d586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.599342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e85c785c-e273-485e-bc46-85ff5ac4d586 None None] MFA Rules not processed for user `3263800cc0234d59a259b690dc68a323`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:20.600614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e85c785c-e273-485e-bc46-85ff5ac4d586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.685589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 948/1928] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:20.690185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.692190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.738598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:20.774336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c866d26-fd99-4b8f-afb2-849d04a3013a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.781661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.782456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.782857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.783224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.784000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.786008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.816007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 981/1929] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:20 2022] POST /identity/v3/projects => generated 389 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:20.820540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c785c-e273-485e-bc46-85ff5ac4d586 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.835015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c785c-e273-485e-bc46-85ff5ac4d586 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.835800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.837754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:20.859284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:20.911625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.912223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.912508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.912785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.913481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.915558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.925638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e27dea-d6da-4915-94d4-0e65479af011 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.930822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.932264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.932556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.932831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] PATH_INFO: `/v3/projects/3004d4d809844046add9180ebf2d7eda` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.934053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3004d4d809844046add9180ebf2d7eda)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.936001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.937897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 949/1930] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:20 2022] POST /identity/v3/users => generated 301 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:20.941914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.951908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ebcffe-28ee-4af9-a42c-75bfc763234b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.956909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:20.957574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:20.957867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:20.958142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:20.958907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:20.960570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:20.970991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 950/1931] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:20.975002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:20.985891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34b277e-e62d-4edd-987a-70d006b80ec7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:20.986420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3004d4d809844046add9180ebf2d7eda', 'request_id': 'req-d6226697-62bf-44b3-8906-c6564658c4b8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:20.995981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3004d4d809844046add9180ebf2d7eda', 'request_id': 'req-d6226697-62bf-44b3-8906-c6564658c4b8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.002813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3004d4d809844046add9180ebf2d7eda', 'request_id': 'req-d6226697-62bf-44b3-8906-c6564658c4b8'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.015240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] The token cache is being invalidate because project 3004d4d809844046add9180ebf2d7eda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:21.015614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.016451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 982/1932] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:20 2022] DELETE /identity/v3/projects/3004d4d809844046add9180ebf2d7eda => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:21.023295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:21.024194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.057194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:21.058134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.058546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.058833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] PATH_INFO: `/v3/projects/49f27bf77799493fa5035029fe7612b5/users/ee26bbef761b4998a4f9b2dc50288e65/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.077188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49f27bf77799493fa5035029fe7612b5, user_id=ee26bbef761b4998a4f9b2dc50288e65, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:21.080092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:21.090105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:21.103372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 951/1933] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:20 2022] PUT /identity/v3/projects/49f27bf77799493fa5035029fe7612b5/users/ee26bbef761b4998a4f9b2dc50288e65/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:21.107681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:21.109422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.114952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6226697-62bf-44b3-8906-c6564658c4b8 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:21.175554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406af395-0b48-409e-9923-940b123df222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:21.176187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced29018-102b-42da-9679-32b67648f0b7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:21.176455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406af395-0b48-409e-9923-940b123df222 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.176744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406af395-0b48-409e-9923-940b123df222 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.177028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406af395-0b48-409e-9923-940b123df222 admin admin] PATH_INFO: `/v3/users/471305aa418a451199ce21b8a201b217` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.178271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406af395-0b48-409e-9923-940b123df222 admin admin] RBAC: Authorizing `identity:delete_user(user_id=471305aa418a451199ce21b8a201b217)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:21.180307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406af395-0b48-409e-9923-940b123df222 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:21.180924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:21.181999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.182169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.182545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.183363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:21.185067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:21.196686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 952/1934] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:21 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:21.201573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:21.213976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74ec969-fdc1-411f-bc1b-ceeeace9bbf2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:21.220095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffdcffa-24c4-452f-8c1a-12c97e014f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:21.221076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffdcffa-24c4-452f-8c1a-12c97e014f69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.221447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffdcffa-24c4-452f-8c1a-12c97e014f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.221872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffdcffa-24c4-452f-8c1a-12c97e014f69 admin admin] PATH_INFO: `/v3/projects/49f27bf77799493fa5035029fe7612b5/users/ee26bbef761b4998a4f9b2dc50288e65/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.224119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffdcffa-24c4-452f-8c1a-12c97e014f69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49f27bf77799493fa5035029fe7612b5, user_id=ee26bbef761b4998a4f9b2dc50288e65, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:21.227146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffdcffa-24c4-452f-8c1a-12c97e014f69 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:21.233401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '471305aa418a451199ce21b8a201b217'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.240171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 953/1935] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:21 2022] PUT /identity/v3/projects/49f27bf77799493fa5035029fe7612b5/users/ee26bbef761b4998a4f9b2dc50288e65/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:21.245696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '471305aa418a451199ce21b8a201b217'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.246365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '471305aa418a451199ce21b8a201b217'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.251376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.251717 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.252004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.255484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '471305aa418a451199ce21b8a201b217'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.270167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '471305aa418a451199ce21b8a201b217'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.272344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:21.279379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 983/1936] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:21 2022] DELETE /identity/v3/users/471305aa418a451199ce21b8a201b217 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:21.292448 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] MFA Rules not processed for user `ee26bbef761b4998a4f9b2dc50288e65`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:21.293819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.402659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 954/1937] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:21 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:21.411316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:21.413476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.486295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406af395-0b48-409e-9923-940b123df222 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:21.489971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:21.491164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.491164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.491437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.502723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:21.504296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:21.518225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 984/1938] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:21 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:21.602770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:21.604386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.674573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:21.675659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.690472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b810087-7856-4346-91d0-782ed14c8a13 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:21.694493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f43557-01a7-435b-8aa0-4f4de2a3487b service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:21.696788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:21.699495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.699766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.700065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] PATH_INFO: `/v3/projects/bd97a99ee0da4c71bfd57552417d956f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.701992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:21.702152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd97a99ee0da4c71bfd57552417d956f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:21.703149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.703542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.703952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] PATH_INFO: `/v3/users/3263800cc0234d59a259b690dc68a323` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.704737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:21.705960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3263800cc0234d59a259b690dc68a323)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:21.708184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:21.721379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:21.770755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3263800cc0234d59a259b690dc68a323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.774674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd97a99ee0da4c71bfd57552417d956f', 'request_id': 'req-be601375-0875-4108-8ecc-15f78398bb87'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.780315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3263800cc0234d59a259b690dc68a323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.781042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3263800cc0234d59a259b690dc68a323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.786523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd97a99ee0da4c71bfd57552417d956f', 'request_id': 'req-be601375-0875-4108-8ecc-15f78398bb87'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.790618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3263800cc0234d59a259b690dc68a323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.797890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd97a99ee0da4c71bfd57552417d956f', 'request_id': 'req-be601375-0875-4108-8ecc-15f78398bb87'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.811841 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9de7dd5d-35eb-4666-b522-69e1feeaf5f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3263800cc0234d59a259b690dc68a323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.812493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] The token cache is being invalidate because project bd97a99ee0da4c71bfd57552417d956f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:21.812909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.813590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-be601375-0875-4108-8ecc-15f78398bb87 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:21.815221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 955/1939] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:21 2022] DELETE /identity/v3/projects/bd97a99ee0da4c71bfd57552417d956f => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:21.821403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b428a7d-c0f3-4a57-a98c-00d838508f6f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.821851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b428a7d-c0f3-4a57-a98c-00d838508f6f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.822140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b428a7d-c0f3-4a57-a98c-00d838508f6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.823103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 985/1940] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:21 2022] DELETE /identity/v3/users/3263800cc0234d59a259b690dc68a323 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:21.828221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.828523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.828807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.851487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:21.851738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-0b428a7d-c0f3-4a57-a98c-00d838508f6f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:21.853046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0b428a7d-c0f3-4a57-a98c-00d838508f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.853268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.890423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:21.948847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 956/1941] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:21.949196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 986/1942] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:21.958731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b847c587-e263-4418-94a9-eb2da0625a8f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:21.959147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b847c587-e263-4418-94a9-eb2da0625a8f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:21.959493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b847c587-e263-4418-94a9-eb2da0625a8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:21.959608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:21.961205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:21.981595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b847c587-e263-4418-94a9-eb2da0625a8f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:21.982834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b847c587-e263-4418-94a9-eb2da0625a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:22.003383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 957/1943] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:22.004319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b847c587-e263-4418-94a9-eb2da0625a8f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.019515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b847c587-e263-4418-94a9-eb2da0625a8f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.027166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.028026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.028405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.028782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.029655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:22.030063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.031822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.040401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaf790a-f8d6-4d81-8a09-7e635dfbb2ad None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.045624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 958/1944] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:22.047735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.048430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.048726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.049269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.049938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.065909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.066147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a56dae-f8d4-4d4d-89c2-7534e08bcbc2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.066558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.076714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.079025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.079606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.080455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.082352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.085004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 987/1945] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:22.088013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.137704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 959/1946] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:22 2022] POST /identity/v3/projects => generated 379 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:22.143255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.156387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44daaa5f-7ec5-43d8-ab66-8e129d6e6848 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.207622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:22.258836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.259763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.260152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.260453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.261044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.263065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.288971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 988/1947] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:22 2022] POST /identity/v3/users => generated 290 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:22.293682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.314226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69cd44-a617-4d2c-ac8d-292753549b7c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.322470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.324921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.324921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.324921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.324921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.326350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.358722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 960/1948] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:22.360259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.371299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab97c95-c147-496e-b25e-aff5a69a40d6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.377409 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca913f53-aa27-49e8-9893-c6bb4172c8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.378097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca913f53-aa27-49e8-9893-c6bb4172c8d7 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.378388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca913f53-aa27-49e8-9893-c6bb4172c8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.378708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca913f53-aa27-49e8-9893-c6bb4172c8d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.379454 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca913f53-aa27-49e8-9893-c6bb4172c8d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.381094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca913f53-aa27-49e8-9893-c6bb4172c8d7 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.391898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 989/1949] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:22.396241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.409791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67107674-1473-40d8-ad8d-dfa505ddd0ec admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.416019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.417924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.418612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.418997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] PATH_INFO: `/v3/projects/0f513abc60d346be8919296c4eb3aa42/users/1b844668de25425a8426a0bee5cbb8d3/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.439556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f513abc60d346be8919296c4eb3aa42, user_id=1b844668de25425a8426a0bee5cbb8d3, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.442909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.458604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 961/1950] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:22 2022] PUT /identity/v3/projects/0f513abc60d346be8919296c4eb3aa42/users/1b844668de25425a8426a0bee5cbb8d3/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:22.471457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeff599-7a97-407f-859b-1eb27146fb18 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.471750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeff599-7a97-407f-859b-1eb27146fb18 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.472031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeff599-7a97-407f-859b-1eb27146fb18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.490955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.492757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:22.503707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-0aeff599-7a97-407f-859b-1eb27146fb18 None None] MFA Rules not processed for user `1b844668de25425a8426a0bee5cbb8d3`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:22.506991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0aeff599-7a97-407f-859b-1eb27146fb18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:22.563914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6392595f-055c-412a-b276-4daf838e42aa admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.568994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.569580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.569865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.570146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] PATH_INFO: `/v3/projects/d96988388648448abbf810754470bda9` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.571508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d96988388648448abbf810754470bda9)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.573958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.580077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 990/1951] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:22 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:22.584931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeff599-7a97-407f-859b-1eb27146fb18 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.596306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeff599-7a97-407f-859b-1eb27146fb18 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.600939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.601606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.601894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.602224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.603419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.605564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.625234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd96988388648448abbf810754470bda9', 'request_id': 'req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.626663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:22.633239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd96988388648448abbf810754470bda9', 'request_id': 'req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.634106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 991/1952] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:22 2022] POST /identity/v3/projects => generated 381 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:22.638930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.645186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd96988388648448abbf810754470bda9', 'request_id': 'req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.667716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] The token cache is being invalidate because project d96988388648448abbf810754470bda9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:22.668340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.669081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.670967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 962/1953] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:22 2022] DELETE /identity/v3/projects/d96988388648448abbf810754470bda9 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:22.675028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.676440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:22.719450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46327bb5-0817-402e-99ad-9ab7f7f177f9 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.725738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.726604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.726784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.727057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.727802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.729939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.754821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 992/1954] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:22 2022] POST /identity/v3/users => generated 297 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:22.755048 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35d15ca-cf3f-4a9d-a10f-25308d87b7f1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.755968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.757509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:22.761483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.762417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.762756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.763034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] PATH_INFO: `/v3/users/ee26bbef761b4998a4f9b2dc50288e65` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.764275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee26bbef761b4998a4f9b2dc50288e65)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.766242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.771451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bbb768-9e54-4770-ad8e-c1ce84ad8cd8 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.777995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.778949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.779179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.780008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.780768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.782419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.783965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:22.807096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 993/1955] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:22.808130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.818682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160df2ba-75ef-47f0-9afb-fc4d4b0e84dd admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:22.823964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:22.824851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:22.825136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:22.825411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] PATH_INFO: `/v3/projects/c1756f9b851c4394b8ba7521da4b01ce/users/55d294d25ec340d1ab1e4d160843de99/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:22.826267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee26bbef761b4998a4f9b2dc50288e65'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.833975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee26bbef761b4998a4f9b2dc50288e65'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.841875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1756f9b851c4394b8ba7521da4b01ce, user_id=55d294d25ec340d1ab1e4d160843de99, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:22.843039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee26bbef761b4998a4f9b2dc50288e65'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.843718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee26bbef761b4998a4f9b2dc50288e65'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.844719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:22.857730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee26bbef761b4998a4f9b2dc50288e65'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:22.858536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 994/1956] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:22 2022] PUT /identity/v3/projects/c1756f9b851c4394b8ba7521da4b01ce/users/55d294d25ec340d1ab1e4d160843de99/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:22.862085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:22.863853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:22.864367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 963/1957] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:22 2022] DELETE /identity/v3/users/ee26bbef761b4998a4f9b2dc50288e65 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:22.953713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987017ec-2866-4786-8c57-2b271499b6a6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.020716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782acc52-9f4a-4810-8c26-88a4f03c073f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.021419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782acc52-9f4a-4810-8c26-88a4f03c073f admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.021886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782acc52-9f4a-4810-8c26-88a4f03c073f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.022171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782acc52-9f4a-4810-8c26-88a4f03c073f admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.022959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782acc52-9f4a-4810-8c26-88a4f03c073f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.024616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782acc52-9f4a-4810-8c26-88a4f03c073f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.038489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 995/1958] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:23.040317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.053795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4441ed9e-1581-4113-a35d-2b3035ed7420 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.060895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.062016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.062315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.062693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] PATH_INFO: `/v3/projects/c1756f9b851c4394b8ba7521da4b01ce/users/55d294d25ec340d1ab1e4d160843de99/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.064846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1756f9b851c4394b8ba7521da4b01ce, user_id=55d294d25ec340d1ab1e4d160843de99, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.067607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.098817 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 964/1959] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:23 2022] PUT /identity/v3/projects/c1756f9b851c4394b8ba7521da4b01ce/users/55d294d25ec340d1ab1e4d160843de99/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:23.105578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.105878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.106167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.141393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] MFA Rules not processed for user `55d294d25ec340d1ab1e4d160843de99`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:23.142650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:23.241578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:23.248675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 996/1960] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:23 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:23.260341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.264868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:23.265458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.267484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:23.318182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:23.333005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b53fe5-5d88-4f36-a62d-50c305c6027c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.337050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aeca12-9abf-4ecc-900b-c0e79694bb8d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.337714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aeca12-9abf-4ecc-900b-c0e79694bb8d service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.338066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aeca12-9abf-4ecc-900b-c0e79694bb8d service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.338335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aeca12-9abf-4ecc-900b-c0e79694bb8d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.343497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4266725-82e5-44d6-85f7-9f493be8b5fc None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.348088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aeca12-9abf-4ecc-900b-c0e79694bb8d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.348952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.349777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aeca12-9abf-4ecc-900b-c0e79694bb8d service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.349908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.350091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.350365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] PATH_INFO: `/v3/projects/49f27bf77799493fa5035029fe7612b5` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.351599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] RBAC: Authorizing `identity:delete_project(project_id=49f27bf77799493fa5035029fe7612b5)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.354085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.360690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 965/1961] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:23 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:23.407002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49f27bf77799493fa5035029fe7612b5', 'request_id': 'req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.415767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49f27bf77799493fa5035029fe7612b5', 'request_id': 'req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.422466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49f27bf77799493fa5035029fe7612b5', 'request_id': 'req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.436163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] The token cache is being invalidate because project 49f27bf77799493fa5035029fe7612b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:23.436455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.437130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77afc55f-fdbd-4db5-8d35-6ec4eaff731a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.438831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 997/1962] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:23 2022] DELETE /identity/v3/projects/49f27bf77799493fa5035029fe7612b5 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:23.443711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e05d02-7736-4d21-b633-a37939d1fa33 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.444012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e05d02-7736-4d21-b633-a37939d1fa33 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.444300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e05d02-7736-4d21-b633-a37939d1fa33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.467345 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-52e05d02-7736-4d21-b633-a37939d1fa33 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:23.468597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-52e05d02-7736-4d21-b633-a37939d1fa33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:23.548348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 966/1963] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:23.556960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4a9e7-6192-473c-8a3d-631ece4ce053 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.557247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4a9e7-6192-473c-8a3d-631ece4ce053 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.557523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4a9e7-6192-473c-8a3d-631ece4ce053 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.581436 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2cc4a9e7-6192-473c-8a3d-631ece4ce053 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:23.582746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2cc4a9e7-6192-473c-8a3d-631ece4ce053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:23.591736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 998/1964] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:23.595979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e05d02-7736-4d21-b633-a37939d1fa33 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.606631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e05d02-7736-4d21-b633-a37939d1fa33 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.611619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.612287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.612580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.612862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.613608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:23.613916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.615559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.628965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 967/1965] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:23.635019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc4a9e7-6192-473c-8a3d-631ece4ce053 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.646655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc4a9e7-6192-473c-8a3d-631ece4ce053 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.652103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.652778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.653069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.653350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.654054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.656017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.693280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 999/1966] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:23 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:23.694422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.705885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b23c6e7-f98d-4d4a-b6d2-97da3b6c4a34 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.779044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.779755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.780051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.780334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.781072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.783726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.806077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.808562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:23.813218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 968/1967] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:23 2022] POST /identity/v3/users => generated 293 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:23.814279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.822963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e833e429-d460-449f-834d-8fa84b728796 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.828264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.829248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.829607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.829986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] PATH_INFO: `/v3/users/1b844668de25425a8426a0bee5cbb8d3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.831633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b844668de25425a8426a0bee5cbb8d3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.832092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a942e4d2-3bdc-47c0-8bab-527d4fe30586 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.833963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.838595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.839547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.839848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.840129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.840878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.842565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.855397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 969/1968] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:23.859869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.870813 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcaa22c-374a-4142-9204-7f285942c7b7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.875930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.876793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.877026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.877306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.878117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.879981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.888484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b844668de25425a8426a0bee5cbb8d3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.891659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 970/1969] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:23.896322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:23.899742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b844668de25425a8426a0bee5cbb8d3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.901047 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b844668de25425a8426a0bee5cbb8d3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.903633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=82,keystone:DELETE=15,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:23.910877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b844668de25425a8426a0bee5cbb8d3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.914876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e2f00c-6ff1-4746-8f27-f0de8b6147dd admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:23.923043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8799c0-4479-4db8-ba27-5de2aed6433c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:23.923953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8799c0-4479-4db8-ba27-5de2aed6433c admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.924241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8799c0-4479-4db8-ba27-5de2aed6433c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8799c0-4479-4db8-ba27-5de2aed6433c admin admin] PATH_INFO: `/v3/projects/5776c0c20ff3473497b5e15a2d167f6b/users/b7e357dc3a2548fc8a3f4ced1d61cbba/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.926961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-df872463-727b-4399-be56-b1ccefd272fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b844668de25425a8426a0bee5cbb8d3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:23.933814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1000/1970] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:23 2022] DELETE /identity/v3/users/1b844668de25425a8426a0bee5cbb8d3 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:23.938364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.938693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.938977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.940688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8799c0-4479-4db8-ba27-5de2aed6433c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5776c0c20ff3473497b5e15a2d167f6b, user_id=b7e357dc3a2548fc8a3f4ced1d61cbba, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:23.943550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8799c0-4479-4db8-ba27-5de2aed6433c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:23.959835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 971/1971] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:23 2022] PUT /identity/v3/projects/5776c0c20ff3473497b5e15a2d167f6b/users/b7e357dc3a2548fc8a3f4ced1d61cbba/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:23.961345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:23.962631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:23.970169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:23.970501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:23.970952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:23.997968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] MFA Rules not processed for user `b7e357dc3a2548fc8a3f4ced1d61cbba`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:23.999296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:24.004564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:24.030642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1001/1972] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:24.040613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:24.042591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:24.074995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 972/1973] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:24.087240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:24.089255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:24.112558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b69bcee-1271-4643-ac7b-8988da88f4e8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:24.116488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:24.117196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:24.117517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:24.117793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:24.128295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:24.129771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:24.140392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1002/1974] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:24.156032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c658159-247a-4e73-b207-e69c448e74ef None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:24.160459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:24.161208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:24.161432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:24.161718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:24.172922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:24.174628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:24.186972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 973/1975] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:24 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:24.607908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:24.610001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:24.628312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307457a-d261-4b66-a4bf-efb4d3ea65f5 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:24.633810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:24.634700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:24.635029 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:24.635373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] PATH_INFO: `/v3/projects/0f513abc60d346be8919296c4eb3aa42` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:24.636679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f513abc60d346be8919296c4eb3aa42)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:24.638739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:24.716729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f513abc60d346be8919296c4eb3aa42', 'request_id': 'req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.727884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f513abc60d346be8919296c4eb3aa42', 'request_id': 'req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.734547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f513abc60d346be8919296c4eb3aa42', 'request_id': 'req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.742541 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:24.746918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] The token cache is being invalidate because project 0f513abc60d346be8919296c4eb3aa42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:24.747533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.748476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.750107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1003/1976] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:24 2022] DELETE /identity/v3/projects/0f513abc60d346be8919296c4eb3aa42 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:24.755469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:24.767110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf267b6c-6f88-40de-bd1a-f00e672c265a service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:24.825575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:24.826355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:24.826685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:24.826975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] PATH_INFO: `/v3/users/55d294d25ec340d1ab1e4d160843de99` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:24.828175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55d294d25ec340d1ab1e4d160843de99)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:24.830127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:24.878108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55d294d25ec340d1ab1e4d160843de99'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.882708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:24.886952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55d294d25ec340d1ab1e4d160843de99'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.894967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55d294d25ec340d1ab1e4d160843de99'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.895625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55d294d25ec340d1ab1e4d160843de99'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.909134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-372cf49b-98ae-4b9d-9c0b-d22620f39a33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55d294d25ec340d1ab1e4d160843de99'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:24.915201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 974/1977] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:24 2022] DELETE /identity/v3/users/55d294d25ec340d1ab1e4d160843de99 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:25.200857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:25.202311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:25.217122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e8448a-ba80-4f96-b0fd-6629e9b82ef3 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:25.279880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:25.280777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.281064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.281349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] PATH_INFO: `/v3/projects/c1756f9b851c4394b8ba7521da4b01ce` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.282776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1756f9b851c4394b8ba7521da4b01ce)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:25.284726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:25.340936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1756f9b851c4394b8ba7521da4b01ce', 'request_id': 'req-2a543c07-c0de-4325-9802-2dae5552c0d2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:25.349749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1756f9b851c4394b8ba7521da4b01ce', 'request_id': 'req-2a543c07-c0de-4325-9802-2dae5552c0d2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:25.378821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1756f9b851c4394b8ba7521da4b01ce', 'request_id': 'req-2a543c07-c0de-4325-9802-2dae5552c0d2'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:25.391562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] The token cache is being invalidate because project c1756f9b851c4394b8ba7521da4b01ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:25.391953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:25.392702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a543c07-c0de-4325-9802-2dae5552c0d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:25.394878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1004/1978] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:25 2022] DELETE /identity/v3/projects/c1756f9b851c4394b8ba7521da4b01ce => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:25.400012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe3687-db7b-4bae-8fa5-a430dbd5b77a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.401015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe3687-db7b-4bae-8fa5-a430dbd5b77a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.401356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe3687-db7b-4bae-8fa5-a430dbd5b77a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.424465 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7afe3687-db7b-4bae-8fa5-a430dbd5b77a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:25.425657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7afe3687-db7b-4bae-8fa5-a430dbd5b77a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:25.505394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 975/1979] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:25.513394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b85f55e-2627-4101-b244-478b8faf1da0 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.513736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b85f55e-2627-4101-b244-478b8faf1da0 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.513971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b85f55e-2627-4101-b244-478b8faf1da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.534087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9b85f55e-2627-4101-b244-478b8faf1da0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:25.535352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9b85f55e-2627-4101-b244-478b8faf1da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:25.546715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1005/1980] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:25.550399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe3687-db7b-4bae-8fa5-a430dbd5b77a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:25.572795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe3687-db7b-4bae-8fa5-a430dbd5b77a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:25.580589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:25.581712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.581712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.581888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.582592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:25.582905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:25.584944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:25.601353 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 976/1981] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:25.606244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b85f55e-2627-4101-b244-478b8faf1da0 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:25.629849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b85f55e-2627-4101-b244-478b8faf1da0 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:25.648356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:25.649041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.649331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.649621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.650370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:25.652335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:25.691883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1006/1982] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:25 2022] POST /identity/v3/projects => generated 383 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:25.695962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:25.707486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e501356-0d1f-4188-9688-8d5b384dc49b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:25.784077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:25.785119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.785730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.786164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.787072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:25.789405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:25.817624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 977/1983] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:25 2022] POST /identity/v3/users => generated 292 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:25.819622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:25.832899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933c129-305f-486f-b3d2-494ddf2a3897 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:25.837595 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:25.838453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.838756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.839022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.839762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:25.841383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:25.852574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1007/1984] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:25.858212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:25.869750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7177eb-24ef-4b04-8882-526ae488496a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:25.876343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e23311b-d963-469c-b99f-6753ed03e921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:25.877770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e23311b-d963-469c-b99f-6753ed03e921 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.879697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e23311b-d963-469c-b99f-6753ed03e921 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.881045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e23311b-d963-469c-b99f-6753ed03e921 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.882289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e23311b-d963-469c-b99f-6753ed03e921 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:25.884878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e23311b-d963-469c-b99f-6753ed03e921 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:25.903409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 978/1985] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:25.904782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:25.915896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e62c61-57fb-4a2a-9aa4-8eb87d478e29 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:25.921221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:25.922116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.922396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.922720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] PATH_INFO: `/v3/projects/c6e9a1e10dc441adb06f2a634bbface0/users/57d705f888b54bb6bf41764e1dcdb1c8/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.939256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6e9a1e10dc441adb06f2a634bbface0, user_id=57d705f888b54bb6bf41764e1dcdb1c8, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:25.942223 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:25.956608 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1008/1986] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:25 2022] PUT /identity/v3/projects/c6e9a1e10dc441adb06f2a634bbface0/users/57d705f888b54bb6bf41764e1dcdb1c8/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:25.967010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0fadf9-08e9-4c55-aca7-fcdfbeefd993 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:25.967310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0fadf9-08e9-4c55-aca7-fcdfbeefd993 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:25.967597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0fadf9-08e9-4c55-aca7-fcdfbeefd993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:25.998177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7d0fadf9-08e9-4c55-aca7-fcdfbeefd993 None None] MFA Rules not processed for user `57d705f888b54bb6bf41764e1dcdb1c8`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:25.999805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7d0fadf9-08e9-4c55-aca7-fcdfbeefd993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:26.027260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:26.071554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 979/1987] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:26.076587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.107275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:26.142739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794de62a-42c0-46c0-9872-3f2529646a8a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.147564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.148238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.148524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.148806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.149511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.151504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.178308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1009/1988] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:26 2022] POST /identity/v3/projects => generated 383 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:26.183280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0fadf9-08e9-4c55-aca7-fcdfbeefd993 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.194384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0fadf9-08e9-4c55-aca7-fcdfbeefd993 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.256743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.257422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.257708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.257986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.258755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.261185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.281407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 980/1989] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:26 2022] POST /identity/v3/users => generated 298 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:26.285593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.296782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc274552-d8be-4b5b-86c2-7107a3c94069 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.301592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.302265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.302582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.302862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.303598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.305222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.317589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1010/1990] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:26.321837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.332322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29c7168-fe1f-4996-b27b-190d385e4db2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.337049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.337936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.338228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.338544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] PATH_INFO: `/v3/projects/7c856c0451764296b55ea44bf9374986/users/719bba11049b4d878af4143cfbac085b/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.353888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c856c0451764296b55ea44bf9374986, user_id=719bba11049b4d878af4143cfbac085b, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.356605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.371247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 981/1991] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:26 2022] PUT /identity/v3/projects/7c856c0451764296b55ea44bf9374986/users/719bba11049b4d878af4143cfbac085b/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:26.375185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.385239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1b67d5-d095-4f75-9592-56a9f62f5931 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.451627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caccf9c6-90bf-40fb-8014-c004dff5b690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.452300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caccf9c6-90bf-40fb-8014-c004dff5b690 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.452585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caccf9c6-90bf-40fb-8014-c004dff5b690 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.452874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caccf9c6-90bf-40fb-8014-c004dff5b690 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.453609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caccf9c6-90bf-40fb-8014-c004dff5b690 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.455533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caccf9c6-90bf-40fb-8014-c004dff5b690 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.466306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1011/1992] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:26.470324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.481767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3ee5e1-1972-4e9f-93ab-ab98245669a9 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.486423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.487335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.487659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.487939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] PATH_INFO: `/v3/projects/7c856c0451764296b55ea44bf9374986/users/719bba11049b4d878af4143cfbac085b/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.490245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c856c0451764296b55ea44bf9374986, user_id=719bba11049b4d878af4143cfbac085b, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.493024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.506293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 982/1993] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:26 2022] PUT /identity/v3/projects/7c856c0451764296b55ea44bf9374986/users/719bba11049b4d878af4143cfbac085b/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:26.516782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3141b8d6-c8dd-4bbc-8a8d-54e87b320957 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.517076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3141b8d6-c8dd-4bbc-8a8d-54e87b320957 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.517358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3141b8d6-c8dd-4bbc-8a8d-54e87b320957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.544840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-3141b8d6-c8dd-4bbc-8a8d-54e87b320957 None None] MFA Rules not processed for user `719bba11049b4d878af4143cfbac085b`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:26.546065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3141b8d6-c8dd-4bbc-8a8d-54e87b320957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:26.619626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1012/1994] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:26 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:26.630948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.632671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:26.698518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93dbb7c-ee8e-4a8e-8f5b-6ebcbe45bde0 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.702216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.702912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.703201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.703485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.713804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.715283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.725724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 983/1995] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:26 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:26.818739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3141b8d6-c8dd-4bbc-8a8d-54e87b320957 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:26.840357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3141b8d6-c8dd-4bbc-8a8d-54e87b320957 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:26.845802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391e0385-1a75-4229-8b67-efcc23640a7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:26.846779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391e0385-1a75-4229-8b67-efcc23640a7e service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:26.847412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391e0385-1a75-4229-8b67-efcc23640a7e service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:26.847862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391e0385-1a75-4229-8b67-efcc23640a7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:26.880337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:26.934718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391e0385-1a75-4229-8b67-efcc23640a7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:26.934867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391e0385-1a75-4229-8b67-efcc23640a7e service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:26.945305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1013/1996] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:27.236865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:27.260618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:27.307459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a3921-0ce7-4546-80db-f932444dd39a service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:27.312225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:27.313095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:27.313379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:27.313656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] PATH_INFO: `/v3/users/57d705f888b54bb6bf41764e1dcdb1c8` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:27.314876 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] RBAC: Authorizing `identity:delete_user(user_id=57d705f888b54bb6bf41764e1dcdb1c8)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:27.316819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:27.365753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57d705f888b54bb6bf41764e1dcdb1c8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.373047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57d705f888b54bb6bf41764e1dcdb1c8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.381005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57d705f888b54bb6bf41764e1dcdb1c8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.381717 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57d705f888b54bb6bf41764e1dcdb1c8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.395618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57d705f888b54bb6bf41764e1dcdb1c8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.402994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 984/1997] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:27 2022] DELETE /identity/v3/users/57d705f888b54bb6bf41764e1dcdb1c8 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:27.407808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:27.408101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:27.408388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:27.428008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:27.429843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:27.493651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1014/1998] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:27.504007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:27.505459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:27.564563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45d1708-54e6-4dea-8ecd-bbf6bae2d29f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:27.568323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:27.568973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:27.569251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:27.569527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:27.579554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:27.581225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:27.590926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 985/1999] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:27.865149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:27.866781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:27.880030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849769c8-4bca-4ce1-bfb6-4061d97c9421 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:27.887225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:27.888396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:27.888890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:27.889208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] PATH_INFO: `/v3/projects/c6e9a1e10dc441adb06f2a634bbface0` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:27.890535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6e9a1e10dc441adb06f2a634bbface0)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:27.892525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:27.957082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6e9a1e10dc441adb06f2a634bbface0', 'request_id': 'req-2ab0290f-e46b-470c-aa65-1d608f0c454e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.964051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6e9a1e10dc441adb06f2a634bbface0', 'request_id': 'req-2ab0290f-e46b-470c-aa65-1d608f0c454e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.970114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6e9a1e10dc441adb06f2a634bbface0', 'request_id': 'req-2ab0290f-e46b-470c-aa65-1d608f0c454e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.982735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] The token cache is being invalidate because project c6e9a1e10dc441adb06f2a634bbface0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:27.983031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.983685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:27.985252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1015/2000] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:27 2022] DELETE /identity/v3/projects/c6e9a1e10dc441adb06f2a634bbface0 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:27.989209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:27.999675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34259660-a785-44d9-a50b-d6cd86e35613 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:28.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:28.063643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:28.064680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:28.064970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:28.065362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] PATH_INFO: `/v3/users/719bba11049b4d878af4143cfbac085b` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:28.066669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] RBAC: Authorizing `identity:delete_user(user_id=719bba11049b4d878af4143cfbac085b)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:28.068644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:28.129253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '719bba11049b4d878af4143cfbac085b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.142969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '719bba11049b4d878af4143cfbac085b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.152803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '719bba11049b4d878af4143cfbac085b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.154470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '719bba11049b4d878af4143cfbac085b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.174157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-001c0a23-5339-44ec-bd19-b47af3b9e18c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '719bba11049b4d878af4143cfbac085b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.182915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 986/2001] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:27 2022] DELETE /identity/v3/users/719bba11049b4d878af4143cfbac085b => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:28.536641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:28.538239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:28.557558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0290f-e46b-470c-aa65-1d608f0c454e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:28.611614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:28.628631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:28.629533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:28.629824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:28.630215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] PATH_INFO: `/v3/projects/7c856c0451764296b55ea44bf9374986` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:28.631833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c856c0451764296b55ea44bf9374986)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:28.633625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:28.687732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c856c0451764296b55ea44bf9374986', 'request_id': 'req-0e454de5-d57f-460d-a537-1686ddfa161c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.696167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c856c0451764296b55ea44bf9374986', 'request_id': 'req-0e454de5-d57f-460d-a537-1686ddfa161c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.701859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c856c0451764296b55ea44bf9374986', 'request_id': 'req-0e454de5-d57f-460d-a537-1686ddfa161c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.713205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] The token cache is being invalidate because project 7c856c0451764296b55ea44bf9374986 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:28.713494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.714152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e454de5-d57f-460d-a537-1686ddfa161c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:28.715771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1016/2002] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:28 2022] DELETE /identity/v3/projects/7c856c0451764296b55ea44bf9374986 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:28.720700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d3621-df5c-4268-afb1-f7c1ddc216aa None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:28.721154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d3621-df5c-4268-afb1-f7c1ddc216aa None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:28.721243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d3621-df5c-4268-afb1-f7c1ddc216aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:28.738961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-949d3621-df5c-4268-afb1-f7c1ddc216aa None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:28.740183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-949d3621-df5c-4268-afb1-f7c1ddc216aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:28.801536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 987/2003] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:28.809577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b03163-b5e0-48b0-ae7f-176ebb54279e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:28.809867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b03163-b5e0-48b0-ae7f-176ebb54279e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:28.810176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b03163-b5e0-48b0-ae7f-176ebb54279e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:28.828912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-52b03163-b5e0-48b0-ae7f-176ebb54279e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:28.830144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-52b03163-b5e0-48b0-ae7f-176ebb54279e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:28.839004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1017/2004] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:28.843497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949d3621-df5c-4268-afb1-f7c1ddc216aa None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:28.854155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949d3621-df5c-4268-afb1-f7c1ddc216aa None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:28.858912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:28.859604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:28.859887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:28.860172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:28.860947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:28.861266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:28.862911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:28.875182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 988/2005] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:28.879744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b03163-b5e0-48b0-ae7f-176ebb54279e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:28.889788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b03163-b5e0-48b0-ae7f-176ebb54279e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:28.894534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:28.895185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:28.895471 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:28.895747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:28.896442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:28.898365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:28.924542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1018/2006] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:28 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:28.928930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:28.940292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776b6b67-3289-4df7-9dab-1582d0d5ca00 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:28.987681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:29.005917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.006637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.006927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.007212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.007933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.010557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.030067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 989/2007] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:28 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:29.034573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.045571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a005fc-3a24-4bb1-9d6d-d422a851c112 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.051469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.052415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.052713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.052994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.053756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.055458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.068087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1019/2008] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:29.072907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.083696 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15180b89-c160-4827-aad4-2ea50d7e03e5 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.088286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f6563d-134d-410a-b0b0-fc852a6ae73c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.088981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f6563d-134d-410a-b0b0-fc852a6ae73c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.089265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f6563d-134d-410a-b0b0-fc852a6ae73c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.089545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f6563d-134d-410a-b0b0-fc852a6ae73c admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.090269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f6563d-134d-410a-b0b0-fc852a6ae73c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.091899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f6563d-134d-410a-b0b0-fc852a6ae73c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.101982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 990/2009] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:29.106588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.117307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8915ec2e-176b-4a6b-a823-a471fb86fd64 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.123801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.124691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.124972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.125249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] PATH_INFO: `/v3/projects/1602720716b94cd6967401f406339126/users/2ca560c0805f4c29a8e8ce025021cb4d/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.140946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1602720716b94cd6967401f406339126, user_id=2ca560c0805f4c29a8e8ce025021cb4d, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.143674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.157455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1020/2010] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:29 2022] PUT /identity/v3/projects/1602720716b94cd6967401f406339126/users/2ca560c0805f4c29a8e8ce025021cb4d/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:29.167666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928df7c4-2cb4-4888-9dc5-1fab076c77b1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.167963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928df7c4-2cb4-4888-9dc5-1fab076c77b1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.168246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928df7c4-2cb4-4888-9dc5-1fab076c77b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.196299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-928df7c4-2cb4-4888-9dc5-1fab076c77b1 None None] MFA Rules not processed for user `2ca560c0805f4c29a8e8ce025021cb4d`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:29.198291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-928df7c4-2cb4-4888-9dc5-1fab076c77b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:29.271155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 991/2011] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:29.275994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.342028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d4d1b3-8db8-412a-a146-34be72052075 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.348826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.350359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.350359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.350359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.350808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.354098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.371011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:29.388612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1021/2012] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:29 2022] POST /identity/v3/projects => generated 383 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:29.392844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928df7c4-2cb4-4888-9dc5-1fab076c77b1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.405807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928df7c4-2cb4-4888-9dc5-1fab076c77b1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.490706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.491208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.491495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.491775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.492475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.494451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.523714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 992/2013] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:29 2022] POST /identity/v3/users => generated 298 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:29.528393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.539118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b6c639-e15e-4539-90af-c44185b4b682 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.544026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.544686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.544975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.545262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.545997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.547680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.559420 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1022/2014] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:29.564138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.575709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c24f80-c203-49ca-b3f3-7edb54ea4341 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.581171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.582127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.582374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.582685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] PATH_INFO: `/v3/projects/33e3c50edbc141f2a26486daa53d0104/users/1b40b4e9bfbb431b8a610eec8bf1c572/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.600647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33e3c50edbc141f2a26486daa53d0104, user_id=1b40b4e9bfbb431b8a610eec8bf1c572, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.604248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.618804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 993/2015] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:29 2022] PUT /identity/v3/projects/33e3c50edbc141f2a26486daa53d0104/users/1b40b4e9bfbb431b8a610eec8bf1c572/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:29.622765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.633193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e0d9d-a0b4-41d4-b83d-37fa4d1283cc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.698448 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa90912-d116-43f7-aa36-ea338b865d80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa90912-d116-43f7-aa36-ea338b865d80 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.699506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa90912-d116-43f7-aa36-ea338b865d80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.699796 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa90912-d116-43f7-aa36-ea338b865d80 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.700571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa90912-d116-43f7-aa36-ea338b865d80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.702370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa90912-d116-43f7-aa36-ea338b865d80 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.713219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1023/2016] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:29 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:29.717786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.728283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57455343-707a-4b44-a80f-303fd267cfe4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.733014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.733943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.734230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.734546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] PATH_INFO: `/v3/projects/33e3c50edbc141f2a26486daa53d0104/users/1b40b4e9bfbb431b8a610eec8bf1c572/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.736540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33e3c50edbc141f2a26486daa53d0104, user_id=1b40b4e9bfbb431b8a610eec8bf1c572, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:29.739318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:29.750612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 994/2017] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:29 2022] PUT /identity/v3/projects/33e3c50edbc141f2a26486daa53d0104/users/1b40b4e9bfbb431b8a610eec8bf1c572/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:29.762544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.762843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.763130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:29.790351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] MFA Rules not processed for user `1b40b4e9bfbb431b8a610eec8bf1c572`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:29.791603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:29.862099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1024/2018] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:29 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:29.873319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:29.875059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:29.886575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:29.936137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3114214f-7006-41aa-8afe-db89b10d8086 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:29.939955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8e24f6-1ae7-483b-981f-a40ee69f11c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:29.940606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8e24f6-1ae7-483b-981f-a40ee69f11c0 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:29.940889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8e24f6-1ae7-483b-981f-a40ee69f11c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:29.941167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8e24f6-1ae7-483b-981f-a40ee69f11c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:30.003505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8e24f6-1ae7-483b-981f-a40ee69f11c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:30.005042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8e24f6-1ae7-483b-981f-a40ee69f11c0 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:30.015819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 995/2019] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:33.053974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:33.055636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:33.165823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9af4474-b9c5-4dca-a06e-52df88af16a5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:33.184114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:33.186161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:33.186724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:33.187970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] PATH_INFO: `/v3/users/b7e357dc3a2548fc8a3f4ced1d61cbba` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:33.191050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7e357dc3a2548fc8a3f4ced1d61cbba)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:33.195756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:33.204073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:33.284148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7e357dc3a2548fc8a3f4ced1d61cbba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.293737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7e357dc3a2548fc8a3f4ced1d61cbba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.294770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7e357dc3a2548fc8a3f4ced1d61cbba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.303126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7e357dc3a2548fc8a3f4ced1d61cbba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.321140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7e357dc3a2548fc8a3f4ced1d61cbba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.332824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1025/2020] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:33 2022] DELETE /identity/v3/users/b7e357dc3a2548fc8a3f4ced1d61cbba => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:33.335133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:33.335539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:33.335750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:33.360524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:33.361841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:33.430628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 996/2021] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:33.443603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:33.445108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:33.511386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f619691f-f2d2-4891-93e7-2c8d70ac1a83 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:33.515607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d709891-42fc-45e8-b3c2-2246772511e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:33.516290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d709891-42fc-45e8-b3c2-2246772511e7 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:33.516579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d709891-42fc-45e8-b3c2-2246772511e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:33.516861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d709891-42fc-45e8-b3c2-2246772511e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:33.527220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d709891-42fc-45e8-b3c2-2246772511e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:33.528620 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d709891-42fc-45e8-b3c2-2246772511e7 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:33.539358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1026/2022] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:33.868035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:33.869923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:33.891197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e0ae62-2066-44db-b285-56638f6bf7bc None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:33.897094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:33.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:33.898785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:33.899246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] PATH_INFO: `/v3/projects/5776c0c20ff3473497b5e15a2d167f6b` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:33.901385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5776c0c20ff3473497b5e15a2d167f6b)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:33.903417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:33.953298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5776c0c20ff3473497b5e15a2d167f6b', 'request_id': 'req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.959263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5776c0c20ff3473497b5e15a2d167f6b', 'request_id': 'req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.961160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:33.969622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5776c0c20ff3473497b5e15a2d167f6b', 'request_id': 'req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.981398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] The token cache is being invalidate because project 5776c0c20ff3473497b5e15a2d167f6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:33.981695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.982365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a69ffae6-c93f-4c2b-a1d6-4713a4bb9c2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:33.983989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 997/2023] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:33 2022] DELETE /identity/v3/projects/5776c0c20ff3473497b5e15a2d167f6b => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:33.989059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da63f88-608f-4666-8421-81868576ca59 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:33.989353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da63f88-608f-4666-8421-81868576ca59 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:33.989633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da63f88-608f-4666-8421-81868576ca59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.007908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-0da63f88-608f-4666-8421-81868576ca59 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:34.009199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0da63f88-608f-4666-8421-81868576ca59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:34.075245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1027/2024] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:34.083492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7a245b-ef62-419c-a6a0-2666bac30493 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.083782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7a245b-ef62-419c-a6a0-2666bac30493 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.084067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7a245b-ef62-419c-a6a0-2666bac30493 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.104140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bd7a245b-ef62-419c-a6a0-2666bac30493 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:34.105354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bd7a245b-ef62-419c-a6a0-2666bac30493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:34.113513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 998/2025] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:34.117868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da63f88-608f-4666-8421-81868576ca59 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.128013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da63f88-608f-4666-8421-81868576ca59 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.132730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.133392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.133679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.133958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.135315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:34.135627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.137265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.149378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1028/2026] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:34.154172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7a245b-ef62-419c-a6a0-2666bac30493 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.165736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7a245b-ef62-419c-a6a0-2666bac30493 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.170883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.171564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.171850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.172200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.172853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.174864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.202585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 999/2027] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:34 2022] POST /identity/v3/projects => generated 399 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:34.207117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.217809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cb42d-2aa0-4818-8b5c-a8453b101d8c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.229903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:34.280243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.280933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.281222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.281513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.282225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.284381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.312123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1029/2028] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:34 2022] POST /identity/v3/users => generated 300 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:34.312261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.324185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608619af-3938-42be-85af-c65a6b557b71 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.329113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.329749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.330036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.330607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.331342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.333089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.348675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1000/2029] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:34.349702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.361032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432285f2-4549-4c88-9f04-2e5c83d348bc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.367804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc5c634-9845-43b3-98b4-2bdd2bb59093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.367804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc5c634-9845-43b3-98b4-2bdd2bb59093 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.367804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc5c634-9845-43b3-98b4-2bdd2bb59093 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.375241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc5c634-9845-43b3-98b4-2bdd2bb59093 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.375610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc5c634-9845-43b3-98b4-2bdd2bb59093 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.377290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc5c634-9845-43b3-98b4-2bdd2bb59093 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.389259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1030/2030] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:34 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:34.393236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.406457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb09eef8-cae1-46d2-ac13-9a5c4b0f2767 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.421268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.422397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.422885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.423406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] PATH_INFO: `/v3/projects/987fe694f65e4a349a4dd5569b9570a8/users/7c4d362ea6a94930b68ff60244509bdd/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.451369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] RBAC: Authorizing `identity:create_grant(project_id=987fe694f65e4a349a4dd5569b9570a8, user_id=7c4d362ea6a94930b68ff60244509bdd, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.454242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.469476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1001/2031] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:34 2022] PUT /identity/v3/projects/987fe694f65e4a349a4dd5569b9570a8/users/7c4d362ea6a94930b68ff60244509bdd/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:34.480808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.481193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.481937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.510873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.513483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:34.515866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] MFA Rules not processed for user `7c4d362ea6a94930b68ff60244509bdd`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:34.517395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:34.591979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a33d8a-3f27-40fb-b348-b2198c2d6471 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.596142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfc5cd6-3336-4c1f-b3fe-230f09b8519b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.597123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfc5cd6-3336-4c1f-b3fe-230f09b8519b service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.597422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfc5cd6-3336-4c1f-b3fe-230f09b8519b service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.597721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfc5cd6-3336-4c1f-b3fe-230f09b8519b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.599984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4dfc5cd6-3336-4c1f-b3fe-230f09b8519b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:34.601045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1031/2032] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:34 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:34.617813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:34.620930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:34.636240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a0c15e-5441-40bf-b610-4147c64bdb52 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:34.641503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18053bc4-d4b5-41a0-b3b7-0630d5a46a01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:34.642173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18053bc4-d4b5-41a0-b3b7-0630d5a46a01 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:34.642632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18053bc4-d4b5-41a0-b3b7-0630d5a46a01 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:34.642919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18053bc4-d4b5-41a0-b3b7-0630d5a46a01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:34.652701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18053bc4-d4b5-41a0-b3b7-0630d5a46a01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.654314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18053bc4-d4b5-41a0-b3b7-0630d5a46a01 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.666137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1032/2033] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:34 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:34.690933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfc5cd6-3336-4c1f-b3fe-230f09b8519b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:34.693198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfc5cd6-3336-4c1f-b3fe-230f09b8519b service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:34.708639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1002/2034] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:34 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:38.495947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a134f8-a160-46d8-a29a-da3258aea690 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:38.496531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a134f8-a160-46d8-a29a-da3258aea690 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:38.496531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a134f8-a160-46d8-a29a-da3258aea690 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:38.497925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1033/2035] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:41:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:41:38.507366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:38.507908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:38.507994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:38.529702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:38.530965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:38.567813 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:INSERT=3,keystone:DELETE=1,keystone:SELECT=96 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:38.570758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a134f8-a160-46d8-a29a-da3258aea690 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:38.572493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-86a134f8-a160-46d8-a29a-da3258aea690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:38.608888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1003/2036] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:41:38 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:38.621451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:38.623241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:38.636830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9308d002-11bb-48d0-be5a-efdfd24a20c1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:38.641848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:38.642526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:38.642814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:38.643093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:38.645190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a134f8-a160-46d8-a29a-da3258aea690 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:38.649791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:38.651217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:38.651504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:38.651782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] PATH_INFO: `/v3/users/85705ea8eb6b43e18d14eb5efefab96f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:38.652352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:38.653010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=85705ea8eb6b43e18d14eb5efefab96f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:38.653888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:38.655002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:38.667113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1004/2037] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:38 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:38.669721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:38.712078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85705ea8eb6b43e18d14eb5efefab96f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:38.721532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85705ea8eb6b43e18d14eb5efefab96f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:38.722369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85705ea8eb6b43e18d14eb5efefab96f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:38.730451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85705ea8eb6b43e18d14eb5efefab96f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:38.745429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-64e390e8-1d7e-4634-9715-3d402b753e7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85705ea8eb6b43e18d14eb5efefab96f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:38.751633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1034/2038] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:38 2022] DELETE /identity/v3/users/85705ea8eb6b43e18d14eb5efefab96f => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:38.934569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:38.945206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b55a4-c3b9-4102-856b-db3995ba5228 service placement] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.009159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.010619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.010908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.011184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] PATH_INFO: `/v3/projects/fd0be78cdcfc4195a0ced5868c14cda5` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.012385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd0be78cdcfc4195a0ced5868c14cda5)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.014411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.061937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd0be78cdcfc4195a0ced5868c14cda5', 'request_id': 'req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:39.067572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd0be78cdcfc4195a0ced5868c14cda5', 'request_id': 'req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:39.075226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd0be78cdcfc4195a0ced5868c14cda5', 'request_id': 'req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:39.086881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] The token cache is being invalidate because project fd0be78cdcfc4195a0ced5868c14cda5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:39.087174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:39.087827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30889ae8-ecec-4d5a-91c2-7b6c9cf603b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:39.089390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1005/2039] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:38 2022] DELETE /identity/v3/projects/fd0be78cdcfc4195a0ced5868c14cda5 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:39.095179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4a839-c09c-4953-987d-bd3feff1d8d3 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.095560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4a839-c09c-4953-987d-bd3feff1d8d3 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.095962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4a839-c09c-4953-987d-bd3feff1d8d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.115098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-67c4a839-c09c-4953-987d-bd3feff1d8d3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:39.116390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-67c4a839-c09c-4953-987d-bd3feff1d8d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:39.175751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1035/2040] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:39.185146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4dfb33-d5b9-43b0-97cb-2163dee02d59 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.185359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4dfb33-d5b9-43b0-97cb-2163dee02d59 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.185643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4dfb33-d5b9-43b0-97cb-2163dee02d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.203474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-5a4dfb33-d5b9-43b0-97cb-2163dee02d59 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:39.204666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-5a4dfb33-d5b9-43b0-97cb-2163dee02d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:39.213097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1006/2041] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:39.217284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c4a839-c09c-4953-987d-bd3feff1d8d3 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.227196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c4a839-c09c-4953-987d-bd3feff1d8d3 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.232390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.233049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.233336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.233618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.234352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:39.234695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.236330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.248953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1036/2042] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:39.253330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4dfb33-d5b9-43b0-97cb-2163dee02d59 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.264365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4dfb33-d5b9-43b0-97cb-2163dee02d59 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.269099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.269763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.270055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.270339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.271067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.273096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.302295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1007/2043] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:39 2022] POST /identity/v3/projects => generated 367 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:39.307211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.325704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a814dac1-7b66-4b14-af30-5e4b2b73c609 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.383366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.384037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.384396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.384662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.385350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.387371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.409824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1037/2044] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:39 2022] POST /identity/v3/users => generated 284 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:39.414108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.425358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cbc1df-8322-4464-b239-054444ae560e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.430259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.430964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.431748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.432031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.432768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.434368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.445206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1008/2045] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:39.449946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.460272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a391f83-53d0-4499-ae99-f10608193cf7 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.465413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f09b5-774c-4d1e-96ee-eb9322769460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.466078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f09b5-774c-4d1e-96ee-eb9322769460 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.466369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f09b5-774c-4d1e-96ee-eb9322769460 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.466699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f09b5-774c-4d1e-96ee-eb9322769460 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.467428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90f09b5-774c-4d1e-96ee-eb9322769460 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.469154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90f09b5-774c-4d1e-96ee-eb9322769460 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.470542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:39.485828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1038/2046] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:39.487112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.498639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e55d59f-468a-49ba-bd39-c58c6537135e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.503402 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.504291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.504572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.504850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] PATH_INFO: `/v3/projects/662b468b634f4ce6871298a131f0c78b/users/40731eeb6998463495012b1a25e39637/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.507236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:39.549751 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] RBAC: Authorizing `identity:create_grant(project_id=662b468b634f4ce6871298a131f0c78b, user_id=40731eeb6998463495012b1a25e39637, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.552667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.571094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1009/2047] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:39 2022] PUT /identity/v3/projects/662b468b634f4ce6871298a131f0c78b/users/40731eeb6998463495012b1a25e39637/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:39.582057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c447999-f9de-4261-9704-3fcb0deb0b04 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.582566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c447999-f9de-4261-9704-3fcb0deb0b04 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.582566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c447999-f9de-4261-9704-3fcb0deb0b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.611934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-6c447999-f9de-4261-9704-3fcb0deb0b04 None None] MFA Rules not processed for user `40731eeb6998463495012b1a25e39637`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:39.613171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6c447999-f9de-4261-9704-3fcb0deb0b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:39.692946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1039/2048] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:39 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:39.694137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.761144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a0b38-1ddc-48ef-9737-4c1f11612859 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.765910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.766609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.766900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.767182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.767884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.769794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.795744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1010/2049] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:39 2022] POST /identity/v3/projects => generated 367 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:39.799987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c447999-f9de-4261-9704-3fcb0deb0b04 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.811198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c447999-f9de-4261-9704-3fcb0deb0b04 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.871279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.871963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.872253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.872538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.873237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.875295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.894526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1040/2050] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:39 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:39.898443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.909840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe9d83c-0a5c-4a62-907b-54d340d903f5 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.914473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.915199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.915978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.916256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.916990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.918744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.929563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1011/2051] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:39.934325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:39.944656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e36fbc-1577-4b12-8192-3fe07d67bfa9 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:39.950960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:39.951907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:39.952223 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:39.952501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] PATH_INFO: `/v3/projects/6d4b2440ade240979fb4b757116390ea/users/5661fa8d42a24a0e8b62d6c0b1b68b4c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:39.969271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d4b2440ade240979fb4b757116390ea, user_id=5661fa8d42a24a0e8b62d6c0b1b68b4c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:39.972000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:39.986280 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1041/2052] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:39 2022] PUT /identity/v3/projects/6d4b2440ade240979fb4b757116390ea/users/5661fa8d42a24a0e8b62d6c0b1b68b4c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:39.990278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:40.001295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd77fc-f9ca-4727-b438-1f45c92a4e86 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:40.059578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a5b633-be65-4033-b659-bfbc1bda165c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:40.060319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a5b633-be65-4033-b659-bfbc1bda165c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:40.060730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a5b633-be65-4033-b659-bfbc1bda165c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:40.061099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a5b633-be65-4033-b659-bfbc1bda165c admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:40.061961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a5b633-be65-4033-b659-bfbc1bda165c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:40.063713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a5b633-be65-4033-b659-bfbc1bda165c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:40.074423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1012/2053] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:40.078822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:40.089390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722daa7d-c146-4544-ab1b-a446d3d11668 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:40.094569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:40.100596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:40.100880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:40.101190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] PATH_INFO: `/v3/projects/6d4b2440ade240979fb4b757116390ea/users/5661fa8d42a24a0e8b62d6c0b1b68b4c/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:40.103952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d4b2440ade240979fb4b757116390ea, user_id=5661fa8d42a24a0e8b62d6c0b1b68b4c, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:40.106812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:40.117677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1042/2054] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:40 2022] PUT /identity/v3/projects/6d4b2440ade240979fb4b757116390ea/users/5661fa8d42a24a0e8b62d6c0b1b68b4c/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:40.128455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:40.128779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:40.129065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:40.156772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] MFA Rules not processed for user `5661fa8d42a24a0e8b62d6c0b1b68b4c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:40.157971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:40.200280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:40.228749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1013/2055] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:40 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:40.240074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:40.241817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:40.285915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:40.303393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2251ac-60ae-4fdd-b949-933ce557a247 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:40.308015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:40.308691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:40.308980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:40.309262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:40.319418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:40.321040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:40.333590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1043/2056] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:40 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:40.416941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:40.418402 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:40.430627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc527585-1b62-49f5-b3be-a5d39a332187 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:40.434187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19d17dd-3a1d-4a16-b79c-de5ea11dcbc6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:40.434868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19d17dd-3a1d-4a16-b79c-de5ea11dcbc6 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:40.435156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19d17dd-3a1d-4a16-b79c-de5ea11dcbc6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:40.435441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19d17dd-3a1d-4a16-b79c-de5ea11dcbc6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:40.492795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19d17dd-3a1d-4a16-b79c-de5ea11dcbc6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:40.494200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19d17dd-3a1d-4a16-b79c-de5ea11dcbc6 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:40.504312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1014/2057] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:40 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:42.090555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:42.092198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:42.188365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3e519-a53f-4169-9318-738a15ee08b5 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:42.196567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:42.197641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:42.198071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:42.198319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] PATH_INFO: `/v3/users/2ca560c0805f4c29a8e8ce025021cb4d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:42.199754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ca560c0805f4c29a8e8ce025021cb4d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:42.201820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:42.257160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ca560c0805f4c29a8e8ce025021cb4d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ca560c0805f4c29a8e8ce025021cb4d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.266153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ca560c0805f4c29a8e8ce025021cb4d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.278315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ca560c0805f4c29a8e8ce025021cb4d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.291567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ca560c0805f4c29a8e8ce025021cb4d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.297861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1044/2058] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:42 2022] DELETE /identity/v3/users/2ca560c0805f4c29a8e8ce025021cb4d => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:42.303018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:42.303552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:42.303641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:42.322062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:42.324497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:42.400968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1015/2059] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:42.410823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:42.412433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:42.476825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72b3ace-f98e-487d-88a0-aef03107caf5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:42.481172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:42.481832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:42.482122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:42.482405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:42.492567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:42.494323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:42.505163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1045/2060] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:42.795389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:42.796848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:42.811073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82977d59-add7-48f7-9256-8d6386722af9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:42.816052 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:42.816944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:42.817226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:42.817553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] PATH_INFO: `/v3/projects/1602720716b94cd6967401f406339126` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:42.818840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=1602720716b94cd6967401f406339126)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:42.820675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:42.868937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1602720716b94cd6967401f406339126', 'request_id': 'req-1079700e-9238-4011-8023-3eac6d250a8f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.874615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1602720716b94cd6967401f406339126', 'request_id': 'req-1079700e-9238-4011-8023-3eac6d250a8f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.882178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1602720716b94cd6967401f406339126', 'request_id': 'req-1079700e-9238-4011-8023-3eac6d250a8f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.890648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:42.894522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] The token cache is being invalidate because project 1602720716b94cd6967401f406339126 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:42.895209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.895873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:42.897426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1016/2061] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:42 2022] DELETE /identity/v3/projects/1602720716b94cd6967401f406339126 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:42.902299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:42.907366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:42.914559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7a500c-4352-47a9-9c9e-befc3bcb4018 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:42.974131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:42.975234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:42.975676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:42.976083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] PATH_INFO: `/v3/users/1b40b4e9bfbb431b8a610eec8bf1c572` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:42.977645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b40b4e9bfbb431b8a610eec8bf1c572)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:42.979898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:43.026864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b40b4e9bfbb431b8a610eec8bf1c572'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.034438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b40b4e9bfbb431b8a610eec8bf1c572'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.035067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b40b4e9bfbb431b8a610eec8bf1c572'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.042527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b40b4e9bfbb431b8a610eec8bf1c572'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.054381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-92b7e06d-2839-44bc-82e2-17de911a6a63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b40b4e9bfbb431b8a610eec8bf1c572'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.060639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1046/2062] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:42 2022] DELETE /identity/v3/users/1b40b4e9bfbb431b8a610eec8bf1c572 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:43.188109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:43.189604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:43.203502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1079700e-9238-4011-8023-3eac6d250a8f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:43.282004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:43.283162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.283512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.283855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] PATH_INFO: `/v3/users/7c4d362ea6a94930b68ff60244509bdd` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.285340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c4d362ea6a94930b68ff60244509bdd)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:43.288234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:43.361312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c4d362ea6a94930b68ff60244509bdd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.370320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c4d362ea6a94930b68ff60244509bdd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.379311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c4d362ea6a94930b68ff60244509bdd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.379929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c4d362ea6a94930b68ff60244509bdd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.394518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c4d362ea6a94930b68ff60244509bdd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.406136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1017/2063] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:43 2022] DELETE /identity/v3/users/7c4d362ea6a94930b68ff60244509bdd => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:43.411090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.411564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.411912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.435579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:43.436858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:43.464021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:43.468591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:43.511282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1047/2064] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:43.526211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:43.528077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:43.559189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd2132a-b8c7-494c-a389-f016e00e4d31 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:43.560797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:43.564331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:43.565320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.565673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.565963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] PATH_INFO: `/v3/projects/33e3c50edbc141f2a26486daa53d0104` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.567288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33e3c50edbc141f2a26486daa53d0104)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:43.569204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:43.597272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:43.601867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238c49c6-117a-4044-8ea9-cb2444091e04 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:43.607439 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c73358-b051-45b8-a771-91544a61de8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:43.608132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c73358-b051-45b8-a771-91544a61de8d service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.608425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c73358-b051-45b8-a771-91544a61de8d service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.608708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c73358-b051-45b8-a771-91544a61de8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.619387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c73358-b051-45b8-a771-91544a61de8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:43.621022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c73358-b051-45b8-a771-91544a61de8d service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:43.629374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33e3c50edbc141f2a26486daa53d0104', 'request_id': 'req-1d3b5230-b611-44b2-bbc2-4be42ae57c40'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.632462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1048/2065] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:43.636051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33e3c50edbc141f2a26486daa53d0104', 'request_id': 'req-1d3b5230-b611-44b2-bbc2-4be42ae57c40'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.644427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33e3c50edbc141f2a26486daa53d0104', 'request_id': 'req-1d3b5230-b611-44b2-bbc2-4be42ae57c40'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.664725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] The token cache is being invalidate because project 33e3c50edbc141f2a26486daa53d0104 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:43.665034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.665771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d3b5230-b611-44b2-bbc2-4be42ae57c40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:43.669368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1018/2066] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:43 2022] DELETE /identity/v3/projects/33e3c50edbc141f2a26486daa53d0104 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:43.676587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e041f138-fe7f-4073-b4d3-b8b113966f24 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.677126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e041f138-fe7f-4073-b4d3-b8b113966f24 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.677673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e041f138-fe7f-4073-b4d3-b8b113966f24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.706839 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e041f138-fe7f-4073-b4d3-b8b113966f24 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:43.708559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e041f138-fe7f-4073-b4d3-b8b113966f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:43.822330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1049/2067] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:43.830784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffd4bf2-a9ef-47dc-8cd7-03592d6dd8d9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.831340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffd4bf2-a9ef-47dc-8cd7-03592d6dd8d9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.832170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffd4bf2-a9ef-47dc-8cd7-03592d6dd8d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.877122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bffd4bf2-a9ef-47dc-8cd7-03592d6dd8d9 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:43.878363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bffd4bf2-a9ef-47dc-8cd7-03592d6dd8d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:43.892849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1019/2068] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:43.893590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e041f138-fe7f-4073-b4d3-b8b113966f24 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:43.905702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e041f138-fe7f-4073-b4d3-b8b113966f24 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:43.910879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:43.911562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.912027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.912274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.919937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:43.919937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:43.919937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:43.929397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1050/2069] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:41:43.934246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffd4bf2-a9ef-47dc-8cd7-03592d6dd8d9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:43.944419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffd4bf2-a9ef-47dc-8cd7-03592d6dd8d9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:43.950048 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:43.950936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:43.951237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:43.951545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:43.952248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:43.954728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:43.991073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1020/2070] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:43 2022] POST /identity/v3/projects => generated 371 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:43.995718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.012013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e6560-9060-4d3f-882e-82827fd54d6c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.032391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.034664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:44.102911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.103703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.103951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.104350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.105602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.108193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.140155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1051/2071] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:43 2022] POST /identity/v3/users => generated 286 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:44.150589 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.150832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ea7c54-a9b7-4009-b76d-7094af5b117b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.163286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.164786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.165255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.165550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] PATH_INFO: `/v3/projects/987fe694f65e4a349a4dd5569b9570a8` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.167587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] RBAC: Authorizing `identity:delete_project(project_id=987fe694f65e4a349a4dd5569b9570a8)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.169774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1cdd33-91ff-4ddc-9fc6-ade5ccbe2dca admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.170639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.185601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.186299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.186609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.186892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.188753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.200324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.224192 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1052/2072] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:44.228579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.244250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '987fe694f65e4a349a4dd5569b9570a8', 'request_id': 'req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:44.252034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '987fe694f65e4a349a4dd5569b9570a8', 'request_id': 'req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:44.253170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5511a92c-7ef3-4569-9e2f-8a23099ff265 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.260462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '987fe694f65e4a349a4dd5569b9570a8', 'request_id': 'req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:44.278743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] The token cache is being invalidate because project 987fe694f65e4a349a4dd5569b9570a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:44.279706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:44.280600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-158dc6c2-dd6a-46a1-af0d-f3f9d72cbbec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:44.282304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1021/2073] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:44 2022] DELETE /identity/v3/projects/987fe694f65e4a349a4dd5569b9570a8 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:44.287189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d10439-fd25-4342-9490-ff769e87145a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.287545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d10439-fd25-4342-9490-ff769e87145a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.288022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d10439-fd25-4342-9490-ff769e87145a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.299027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:44.318963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f6d10439-fd25-4342-9490-ff769e87145a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:44.320289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f6d10439-fd25-4342-9490-ff769e87145a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:44.328375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:44.344065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.345023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.345370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.345910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.347038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.348767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.367080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1053/2074] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:44.368097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.370070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:44.383737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02322d3f-cdef-4622-a2c0-e5df745e51cc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.389428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a17e1-2fc3-4c91-b1c0-9f1082869401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.390414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a17e1-2fc3-4c91-b1c0-9f1082869401 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.390804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a17e1-2fc3-4c91-b1c0-9f1082869401 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.391162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a17e1-2fc3-4c91-b1c0-9f1082869401 admin admin] PATH_INFO: `/v3/projects/9e1006c0d7b44dc58cbb7786d7a78213/users/777d5ea61dea485b80434a1fc030b283/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.395167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1022/2075] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:44.404391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ce2e8-1c94-49b6-b0be-ad9d84288f4c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.404742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ce2e8-1c94-49b6-b0be-ad9d84288f4c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.405086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ce2e8-1c94-49b6-b0be-ad9d84288f4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.411472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a17e1-2fc3-4c91-b1c0-9f1082869401 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e1006c0d7b44dc58cbb7786d7a78213, user_id=777d5ea61dea485b80434a1fc030b283, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.414793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a17e1-2fc3-4c91-b1c0-9f1082869401 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.429307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-370ce2e8-1c94-49b6-b0be-ad9d84288f4c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:44.430803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-370ce2e8-1c94-49b6-b0be-ad9d84288f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:44.431969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1054/2076] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:44 2022] PUT /identity/v3/projects/9e1006c0d7b44dc58cbb7786d7a78213/users/777d5ea61dea485b80434a1fc030b283/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:44.442987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68224081-6cc2-4348-b416-e257df85d53b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.443236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68224081-6cc2-4348-b416-e257df85d53b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.443522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68224081-6cc2-4348-b416-e257df85d53b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.474843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-68224081-6cc2-4348-b416-e257df85d53b None None] MFA Rules not processed for user `777d5ea61dea485b80434a1fc030b283`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:44.476572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-68224081-6cc2-4348-b416-e257df85d53b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:44.501797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1023/2077] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:44.506621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ce2e8-1c94-49b6-b0be-ad9d84288f4c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.529375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ce2e8-1c94-49b6-b0be-ad9d84288f4c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.534669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.535361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.535808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.536092 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.536840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:44.537161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.538842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.554464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1024/2078] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:41:44.558417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.568343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c40cda-9516-4b2d-afac-a0401ae2fd87 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.570060 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1055/2079] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:44.574037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.574744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.575028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.575217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68224081-6cc2-4348-b416-e257df85d53b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.575319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.575991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.577941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.586230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68224081-6cc2-4348-b416-e257df85d53b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.591601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.592266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.592555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.592859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.593574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.595567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.609666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1025/2080] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:44.610741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.630258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1056/2081] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:44.631398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.685481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679c23b6-9fc1-4c40-994b-6cb492f0fe3b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.706766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:44.707302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c465125-e563-49e8-96a2-e58749b3f729 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.713967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.715233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.715598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.716377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.717095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.719916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.737826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:44.748096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1057/2082] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/users => generated 294 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:44.748271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.748787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.749073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.749183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.749355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.750164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.752199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.758922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85df258-fdbe-480f-9a92-51af4f9502c8 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.765561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.766229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.766542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.766820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.767554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.769152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.777396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1026/2083] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/users => generated 289 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:44.778375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.785830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1058/2084] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:44.786869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.788763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065cdf74-04e3-418f-9d15-388d283ed680 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.794210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.794925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.795220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.795506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.796253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.797129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eae58e-b403-46fa-9e94-bd3db0a2c61b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.797875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.801752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.802645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.802928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.803204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] PATH_INFO: `/v3/projects/764ebd462a2b4392b9b0c7ce86c5456e/users/6bec97e7cee94e72ab43370f07088d02/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.808168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1027/2085] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:44.812106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.819579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=764ebd462a2b4392b9b0c7ce86c5456e, user_id=6bec97e7cee94e72ab43370f07088d02, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.822142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b074311-f936-41ac-a3ee-3353fb1e52d5 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.822387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.827074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.827846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.828210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.828512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.829321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.831426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.836022 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1059/2086] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:44 2022] PUT /identity/v3/projects/764ebd462a2b4392b9b0c7ce86c5456e/users/6bec97e7cee94e72ab43370f07088d02/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:44.840757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.845535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1028/2087] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:44.846462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.856659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673394e-2040-4d4f-8158-0efff63855a4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.907924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b0c9f2-0e2d-4943-8365-ccffaa948a9c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.912731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.913407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.913566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b520f34c-0a16-4271-a12c-0a0ecb8c1906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.913704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.913962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.914495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b520f34c-0a16-4271-a12c-0a0ecb8c1906 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.914729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.914832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b520f34c-0a16-4271-a12c-0a0ecb8c1906 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.915054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b520f34c-0a16-4271-a12c-0a0ecb8c1906 admin admin] PATH_INFO: `/v3/projects/3d541b7a72dc4b4494a42bda6fa75026/users/0f8e300cd96e446db7430044e69d997a/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.917014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.929739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1060/2088] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:44 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:44.931614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:44.933997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b520f34c-0a16-4271-a12c-0a0ecb8c1906 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d541b7a72dc4b4494a42bda6fa75026, user_id=0f8e300cd96e446db7430044e69d997a, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.936741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b520f34c-0a16-4271-a12c-0a0ecb8c1906 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.941942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a72b3e3-65cf-42f1-92a1-8a7aa539b02a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:44.947549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e233e2d-097e-4ef0-8265-089b1d2f1571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:44.948597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e233e2d-097e-4ef0-8265-089b1d2f1571 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.948826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e233e2d-097e-4ef0-8265-089b1d2f1571 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.949101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e233e2d-097e-4ef0-8265-089b1d2f1571 admin admin] PATH_INFO: `/v3/projects/764ebd462a2b4392b9b0c7ce86c5456e/users/6bec97e7cee94e72ab43370f07088d02/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.950771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1029/2089] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:44 2022] PUT /identity/v3/projects/3d541b7a72dc4b4494a42bda6fa75026/users/0f8e300cd96e446db7430044e69d997a/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:44.951178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e233e2d-097e-4ef0-8265-089b1d2f1571 admin admin] RBAC: Authorizing `identity:create_grant(project_id=764ebd462a2b4392b9b0c7ce86c5456e, user_id=6bec97e7cee94e72ab43370f07088d02, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:44.954085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e233e2d-097e-4ef0-8265-089b1d2f1571 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:44.956114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:44.956483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:44.956950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:44.966003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1061/2090] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:44 2022] PUT /identity/v3/projects/764ebd462a2b4392b9b0c7ce86c5456e/users/6bec97e7cee94e72ab43370f07088d02/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:44.979461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:44.980747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:45.037700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:45.038079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:45.038177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:45.054801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1030/2091] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:45.065551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:45.067093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:45.067611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] MFA Rules not processed for user `6bec97e7cee94e72ab43370f07088d02`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:45.068850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:45.086188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:45.133051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971d5661-debd-4c45-9c0e-e45e358a0788 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:45.137261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:45.137946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:45.138235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:45.138550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:45.146011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1062/2092] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:45 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:45.148282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:45.150777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:45.157500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:45.158925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:45.161391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1031/2093] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:45.195010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:45.220277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bac311-94e0-4416-b2d4-ed234cb5398e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:45.224340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1485f86c-1fb5-4bc0-a554-d4f721a36382 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:45.225097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485f86c-1fb5-4bc0-a554-d4f721a36382 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:45.225390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485f86c-1fb5-4bc0-a554-d4f721a36382 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:45.225670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485f86c-1fb5-4bc0-a554-d4f721a36382 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:45.297878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1485f86c-1fb5-4bc0-a554-d4f721a36382 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:45.299383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1485f86c-1fb5-4bc0-a554-d4f721a36382 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:45.309122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1063/2094] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:45 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:45.811620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:45.813105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:45.829929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec72857-ffdb-44e8-aa0e-d34a3f9b6c2f service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:45.834370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:45.835318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:45.835607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:45.836096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] PATH_INFO: `/v3/users/40731eeb6998463495012b1a25e39637` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:45.837226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] RBAC: Authorizing `identity:delete_user(user_id=40731eeb6998463495012b1a25e39637)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:45.839774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:45.890847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40731eeb6998463495012b1a25e39637'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:45.898157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40731eeb6998463495012b1a25e39637'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:45.906825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40731eeb6998463495012b1a25e39637'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:45.907506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40731eeb6998463495012b1a25e39637'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:45.920230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40731eeb6998463495012b1a25e39637'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:45.927602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1032/2095] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:45 2022] DELETE /identity/v3/users/40731eeb6998463495012b1a25e39637 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:45.932023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:45.932530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:45.932735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:45.954827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:45.957183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:46.039013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1064/2096] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:46.048209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:46.049817 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:46.123255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a201420-2d56-4d7d-8ac3-ca4722d9258c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:46.127448 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:46.128127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:46.128440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:46.128695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:46.139698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:46.141170 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:46.152326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1033/2097] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:46.470931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:46.473023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:46.486663 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832cca62-777d-48b2-8c47-7c17ba1e2b53 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:46.491654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:46.492832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:46.493124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:46.494145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] PATH_INFO: `/v3/projects/662b468b634f4ce6871298a131f0c78b` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:46.495636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] RBAC: Authorizing `identity:delete_project(project_id=662b468b634f4ce6871298a131f0c78b)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:46.497524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:46.559590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '662b468b634f4ce6871298a131f0c78b', 'request_id': 'req-ebb9c5cf-48b3-4918-8031-6170803d4568'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.569254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '662b468b634f4ce6871298a131f0c78b', 'request_id': 'req-ebb9c5cf-48b3-4918-8031-6170803d4568'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.579028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '662b468b634f4ce6871298a131f0c78b', 'request_id': 'req-ebb9c5cf-48b3-4918-8031-6170803d4568'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.593958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] The token cache is being invalidate because project 662b468b634f4ce6871298a131f0c78b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:46.594340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.595113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.600348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1065/2098] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:46 2022] DELETE /identity/v3/projects/662b468b634f4ce6871298a131f0c78b => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:46.601725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:46.615726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9db123-3eae-45b7-a718-14e24705a766 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:46.659644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:46.697749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:46.698674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:46.698960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:46.699238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] PATH_INFO: `/v3/users/5661fa8d42a24a0e8b62d6c0b1b68b4c` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:46.700482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5661fa8d42a24a0e8b62d6c0b1b68b4c)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:46.702449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:46.806067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5661fa8d42a24a0e8b62d6c0b1b68b4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.815550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5661fa8d42a24a0e8b62d6c0b1b68b4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.824252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5661fa8d42a24a0e8b62d6c0b1b68b4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.825255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5661fa8d42a24a0e8b62d6c0b1b68b4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.843719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9c37c65a-e156-4c2c-b5b5-a455c935da02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5661fa8d42a24a0e8b62d6c0b1b68b4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:46.854472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1034/2099] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:46 2022] DELETE /identity/v3/users/5661fa8d42a24a0e8b62d6c0b1b68b4c => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:47.294129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:47.295788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:47.311718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9c5cf-48b3-4918-8031-6170803d4568 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:47.322703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:47.381931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:47.382780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:47.383110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:47.383396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] PATH_INFO: `/v3/projects/6d4b2440ade240979fb4b757116390ea` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:47.384713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d4b2440ade240979fb4b757116390ea)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:47.386717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:47.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d4b2440ade240979fb4b757116390ea', 'request_id': 'req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:47.457618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d4b2440ade240979fb4b757116390ea', 'request_id': 'req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:47.464953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d4b2440ade240979fb4b757116390ea', 'request_id': 'req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:47.486251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] The token cache is being invalidate because project 6d4b2440ade240979fb4b757116390ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:47.486251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:47.486697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0e72f4e2-114e-4d16-b9c0-a249cc883e5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:47.488566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1066/2100] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:47 2022] DELETE /identity/v3/projects/6d4b2440ade240979fb4b757116390ea => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:47.493620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034ba353-50da-461e-956d-5852d7554d3e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:47.493810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034ba353-50da-461e-956d-5852d7554d3e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:47.494214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034ba353-50da-461e-956d-5852d7554d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:47.530010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-034ba353-50da-461e-956d-5852d7554d3e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:47.531383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-034ba353-50da-461e-956d-5852d7554d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:47.635452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1035/2101] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:47.642893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f32839-3234-48c0-bc1c-f6fa3e535e38 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:47.643418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f32839-3234-48c0-bc1c-f6fa3e535e38 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:47.644108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f32839-3234-48c0-bc1c-f6fa3e535e38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:47.670683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f1f32839-3234-48c0-bc1c-f6fa3e535e38 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:47.672626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f1f32839-3234-48c0-bc1c-f6fa3e535e38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:47.685633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1067/2102] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:47.689484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034ba353-50da-461e-956d-5852d7554d3e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:47.701734 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034ba353-50da-461e-956d-5852d7554d3e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:47.709060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:47.709758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:47.710043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:47.710325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:47.712242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:47.712561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:47.714766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:47.737509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1036/2103] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:41:47.738829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f32839-3234-48c0-bc1c-f6fa3e535e38 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:47.749808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f32839-3234-48c0-bc1c-f6fa3e535e38 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:47.756773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:47.757743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:47.758313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:47.758821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:47.760378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:47.762944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:47.805557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1068/2104] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:47 2022] POST /identity/v3/projects => generated 383 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:47.810332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:47.822378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f83160a-1f2a-45df-9497-ecc444e0f492 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:47.882894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:47.912401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:47.913482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:47.913482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:47.914001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:47.914924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:47.917023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:47.952635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1037/2105] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:47 2022] POST /identity/v3/users => generated 292 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:47.954217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:47.965923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6130376-0b15-4d3a-9f76-5b53fa781328 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:47.971303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:47.972470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:47.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:47.973278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:47.974128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:47.976794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:47.994480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1069/2106] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:47.996930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:48.009151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cf75b6-d4b6-4916-a1ec-9c19ed51e46d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:48.014647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dcad36-6263-404a-9802-db9ec8d40c3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:48.016240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dcad36-6263-404a-9802-db9ec8d40c3c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:48.016240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dcad36-6263-404a-9802-db9ec8d40c3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:48.016240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dcad36-6263-404a-9802-db9ec8d40c3c admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:48.017395 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20dcad36-6263-404a-9802-db9ec8d40c3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:48.019497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20dcad36-6263-404a-9802-db9ec8d40c3c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:48.038526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1038/2107] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:48.041082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:48.052232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a25636-79c7-4396-af1c-52c6cd43a9a4 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:48.058085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:48.059053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:48.059412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:48.059760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] PATH_INFO: `/v3/projects/99111f25c77c4dad81094c50e77a7e16/users/6faec6a1102a44a09062cf3dc6dd5e7b/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:48.077216 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] RBAC: Authorizing `identity:create_grant(project_id=99111f25c77c4dad81094c50e77a7e16, user_id=6faec6a1102a44a09062cf3dc6dd5e7b, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:48.079967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:48.101896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1070/2108] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:48 2022] PUT /identity/v3/projects/99111f25c77c4dad81094c50e77a7e16/users/6faec6a1102a44a09062cf3dc6dd5e7b/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:48.112647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f1347-0dfb-459a-9546-7e008cfaa209 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:48.112918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f1347-0dfb-459a-9546-7e008cfaa209 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:48.113208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f1347-0dfb-459a-9546-7e008cfaa209 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:48.143759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bc9f1347-0dfb-459a-9546-7e008cfaa209 None None] MFA Rules not processed for user `6faec6a1102a44a09062cf3dc6dd5e7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:48.146513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bc9f1347-0dfb-459a-9546-7e008cfaa209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:48.248167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1039/2109] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:48.261700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:48.263829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:48.328799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:48.371463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a716aaa-28b5-4b8d-8923-abb1f7640026 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:48.377430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6296760-02c9-4274-8f29-9186e8adb25b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:48.378110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6296760-02c9-4274-8f29-9186e8adb25b service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:48.378395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6296760-02c9-4274-8f29-9186e8adb25b service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:48.378698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6296760-02c9-4274-8f29-9186e8adb25b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:48.407421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6296760-02c9-4274-8f29-9186e8adb25b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:48.409114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6296760-02c9-4274-8f29-9186e8adb25b service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:48.423603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1071/2110] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:48 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:48.444251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f1347-0dfb-459a-9546-7e008cfaa209 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:48.540006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f1347-0dfb-459a-9546-7e008cfaa209 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:48.545328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:48.546135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:48.546518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:48.546811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] PATH_INFO: `/v3/users/6faec6a1102a44a09062cf3dc6dd5e7b` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:48.548100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=6faec6a1102a44a09062cf3dc6dd5e7b)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:48.550241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:48.609213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6faec6a1102a44a09062cf3dc6dd5e7b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:48.612346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:48.620450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6faec6a1102a44a09062cf3dc6dd5e7b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:48.631965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6faec6a1102a44a09062cf3dc6dd5e7b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:48.632883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6faec6a1102a44a09062cf3dc6dd5e7b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:48.651271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6faec6a1102a44a09062cf3dc6dd5e7b'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:48.665608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1040/2111] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:48 2022] DELETE /identity/v3/users/6faec6a1102a44a09062cf3dc6dd5e7b => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:48.675699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:48.693715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:48.693715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:48.730247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:48.734007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:48.820300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1072/2112] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:48.827748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:48.829836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:48.959325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d0058-df71-4168-ac30-dc6fc17b0e0c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:48.970724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573a053a-7ea6-4e4e-a659-a85b3f4f9519 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:48.971802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a053a-7ea6-4e4e-a659-a85b3f4f9519 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:48.971802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a053a-7ea6-4e4e-a659-a85b3f4f9519 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:48.972083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573a053a-7ea6-4e4e-a659-a85b3f4f9519 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:48.998301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573a053a-7ea6-4e4e-a659-a85b3f4f9519 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:49.000362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573a053a-7ea6-4e4e-a659-a85b3f4f9519 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:49.035298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1041/2113] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:49.470403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:49.473684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:49.513930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4181973e-1ed1-46d0-b6ed-ff435980fa06 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:49.531522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:49.533317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:49.533882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:49.534470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] PATH_INFO: `/v3/projects/99111f25c77c4dad81094c50e77a7e16` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:49.537152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] RBAC: Authorizing `identity:delete_project(project_id=99111f25c77c4dad81094c50e77a7e16)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:49.544756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:49.631508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '99111f25c77c4dad81094c50e77a7e16', 'request_id': 'req-3fb291ff-e6ef-41df-9019-064e1379334d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:49.651702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '99111f25c77c4dad81094c50e77a7e16', 'request_id': 'req-3fb291ff-e6ef-41df-9019-064e1379334d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:49.660403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '99111f25c77c4dad81094c50e77a7e16', 'request_id': 'req-3fb291ff-e6ef-41df-9019-064e1379334d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:49.676310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] The token cache is being invalidate because project 99111f25c77c4dad81094c50e77a7e16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:49.676516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:49.677219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-3fb291ff-e6ef-41df-9019-064e1379334d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:49.680763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1073/2114] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:49 2022] DELETE /identity/v3/projects/99111f25c77c4dad81094c50e77a7e16 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:49.685555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6be3d27-69a1-47b1-9ab9-65700703dd80 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:49.686077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6be3d27-69a1-47b1-9ab9-65700703dd80 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:49.686578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6be3d27-69a1-47b1-9ab9-65700703dd80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:49.709392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d6be3d27-69a1-47b1-9ab9-65700703dd80 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:49.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d6be3d27-69a1-47b1-9ab9-65700703dd80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:49.807732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1042/2115] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:49.816774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74645241-8b45-4fa4-b21e-9fceb4d37a92 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:49.817361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74645241-8b45-4fa4-b21e-9fceb4d37a92 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:49.818050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74645241-8b45-4fa4-b21e-9fceb4d37a92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:49.847009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-74645241-8b45-4fa4-b21e-9fceb4d37a92 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:49.848310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-74645241-8b45-4fa4-b21e-9fceb4d37a92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:49.864155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1074/2116] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:49.870469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6be3d27-69a1-47b1-9ab9-65700703dd80 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:49.879510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6be3d27-69a1-47b1-9ab9-65700703dd80 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:49.884958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2230e135-677e-49d8-a4a9-aa93e8adce22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:49.885636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2230e135-677e-49d8-a4a9-aa93e8adce22 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:49.885924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2230e135-677e-49d8-a4a9-aa93e8adce22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:49.886204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2230e135-677e-49d8-a4a9-aa93e8adce22 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:49.887033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2230e135-677e-49d8-a4a9-aa93e8adce22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:49.887297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2230e135-677e-49d8-a4a9-aa93e8adce22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:49.888913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2230e135-677e-49d8-a4a9-aa93e8adce22 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:49.904644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1043/2117] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:41:49.909036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74645241-8b45-4fa4-b21e-9fceb4d37a92 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:49.920897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74645241-8b45-4fa4-b21e-9fceb4d37a92 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:49.928010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:49.928737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:49.929033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:49.929773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:49.930626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:49.933288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:49.948593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:49.949001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:49.949403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:49.979838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1075/2118] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:49 2022] POST /identity/v3/projects => generated 373 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:49.980226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] MFA Rules not processed for user `0f8e300cd96e446db7430044e69d997a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:49.981404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:49.983487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.041017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:50.043031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:50.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1044/2119] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:49 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:50.077527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.079023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:50.079375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c18746-7f53-4c48-9c34-94b196181023 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.086519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.087354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.087739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.087914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.088687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.090820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.103364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e55f4f-f5e1-49dc-a0ed-a3328747f59c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.112872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.113670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.117194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.117346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1076/2120] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:49 2022] POST /identity/v3/users => generated 287 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:50.117454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.117454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.119088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.119237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.129825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28288d40-1f8c-4b8b-b2e8-ac4fa00386eb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.135236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.135922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.136206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.136485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.137220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.138893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.151304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1077/2121] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:50.155424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.166606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1045/2122] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:50 2022] POST /identity/v3/projects => generated 379 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:50.168352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.169250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9942d466-afd5-46ec-8b81-1e669663ee9e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.240187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.240187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.240187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.240187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.240517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.241431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.260111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1078/2123] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:50.262157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.270629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa004215-8a31-44e0-b137-bf6561529380 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.277844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd061bb-2029-4b28-81fd-281d1ce2550a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.278451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.279387 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.279936 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.280270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.281047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.284260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.286074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.287093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.287453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.287828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] PATH_INFO: `/v3/projects/e33331b4e057438a9aa0bbcffc23f089/users/eac3714a97064436be7501249aba35eb/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.310464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e33331b4e057438a9aa0bbcffc23f089, user_id=eac3714a97064436be7501249aba35eb, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.311902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1046/2124] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:50 2022] POST /identity/v3/users => generated 296 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:50.313482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.313689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.316334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:50.330659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1079/2125] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:50 2022] PUT /identity/v3/projects/e33331b4e057438a9aa0bbcffc23f089/users/eac3714a97064436be7501249aba35eb/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:50.331603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.335999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3b90c2-01d1-498e-92b4-7ef8045343e1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.428154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.429063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.429351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.429636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] PATH_INFO: `/v3/users/777d5ea61dea485b80434a1fc030b283` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.430935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] RBAC: Authorizing `identity:delete_user(user_id=777d5ea61dea485b80434a1fc030b283)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.432891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.434655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c500f0-4d57-41b8-a707-949c6b92e337 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.449128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6862de-a6d7-436d-a4ec-e4ccdfd99391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.449951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6862de-a6d7-436d-a4ec-e4ccdfd99391 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.450250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6862de-a6d7-436d-a4ec-e4ccdfd99391 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.450570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6862de-a6d7-436d-a4ec-e4ccdfd99391 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.451339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6862de-a6d7-436d-a4ec-e4ccdfd99391 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.453277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6862de-a6d7-436d-a4ec-e4ccdfd99391 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.465650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1080/2126] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:50.467077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa922da-91fe-4e90-94df-8a805e1f28b9 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.467385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa922da-91fe-4e90-94df-8a805e1f28b9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.467758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa922da-91fe-4e90-94df-8a805e1f28b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.501858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-8aa922da-91fe-4e90-94df-8a805e1f28b9 None None] MFA Rules not processed for user `eac3714a97064436be7501249aba35eb`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:50.503406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8aa922da-91fe-4e90-94df-8a805e1f28b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:50.510831 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '777d5ea61dea485b80434a1fc030b283'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:50.514975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:50.521782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '777d5ea61dea485b80434a1fc030b283'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:50.523853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:50.539699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '777d5ea61dea485b80434a1fc030b283'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:50.542363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '777d5ea61dea485b80434a1fc030b283'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:50.559350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '777d5ea61dea485b80434a1fc030b283'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:50.566823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1047/2127] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:50 2022] DELETE /identity/v3/users/777d5ea61dea485b80434a1fc030b283 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:50.567530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.570540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:50.587969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1081/2128] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:50 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:50.589201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.589496 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.589840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.608227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:50.609459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:50.662021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b666c10-08b6-45da-805e-4e516f42999b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.667997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.668988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.669281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.669575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] PATH_INFO: `/v3/projects/569751527e5c4587b69f8bee9b359fc2/users/d652ee5c440b41c980191f5c0c1eeb8e/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.682604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1082/2129] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:50.683822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.685244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:50.691727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] RBAC: Authorizing `identity:create_grant(project_id=569751527e5c4587b69f8bee9b359fc2, user_id=d652ee5c440b41c980191f5c0c1eeb8e, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.694871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.699059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2862a3-71c1-4138-9001-c9ca265063f4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.704012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.704784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.705077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.705667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.706312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.708308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.713511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1048/2130] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:50 2022] PUT /identity/v3/projects/569751527e5c4587b69f8bee9b359fc2/users/d652ee5c440b41c980191f5c0c1eeb8e/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:50.716472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.716472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:50.742063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1083/2131] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:50 2022] POST /identity/v3/projects => generated 377 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:50.742934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.781964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f24f6-a5ef-41f2-af22-b7b3bd05d385 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.812506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e912aa-3b29-481f-9cc2-8f041440957d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.817598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.818465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.818763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.819046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.819795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.821490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.839053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1084/2132] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:50.839053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.851966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441da580-56dd-4f35-81ef-bc0df1ec9ed8 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.852725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.853476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.853837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.854840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.857429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.858156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.858551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.858911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.859696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.861917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.869628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.872215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.891031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1049/2133] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:50.892124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.911389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a71eb-ea7e-4989-97cb-a62cf6d2286f service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.917743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:50.919067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:50.919360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:50.919751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1085/2134] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:50 2022] POST /identity/v3/users => generated 289 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:50.920063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] PATH_INFO: `/v3/projects/569751527e5c4587b69f8bee9b359fc2/users/d652ee5c440b41c980191f5c0c1eeb8e/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:50.922123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=569751527e5c4587b69f8bee9b359fc2, user_id=d652ee5c440b41c980191f5c0c1eeb8e, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:50.923363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:50.925017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:50.936442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7445ed-0390-4329-a627-a2e253a629ee admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:50.947737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:50.951083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1050/2135] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:50 2022] PUT /identity/v3/projects/569751527e5c4587b69f8bee9b359fc2/users/d652ee5c440b41c980191f5c0c1eeb8e/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:51.017853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.018923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.019445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.019886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.020891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.023012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.036811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1086/2136] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:51.041490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.047788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:51.058228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bd98eb-8821-4697-b635-d695b3e650b4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:51.068033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2854354d-7a93-426b-b156-bb510881e9df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.068894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854354d-7a93-426b-b156-bb510881e9df admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.069526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854354d-7a93-426b-b156-bb510881e9df admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.069924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854354d-7a93-426b-b156-bb510881e9df admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.072868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854354d-7a93-426b-b156-bb510881e9df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.074761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854354d-7a93-426b-b156-bb510881e9df admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.096578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1051/2137] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:51.097769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.116522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e889a50f-f4a9-4109-acd4-3e21ae0dfd7b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:51.123139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.124307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.124597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.125073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] PATH_INFO: `/v3/projects/4d0078939cee4ad3a68c82e37a4c68f1/users/ab574d9ebed3485587512b2f82d60dc0/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.145882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d0078939cee4ad3a68c82e37a4c68f1, user_id=ab574d9ebed3485587512b2f82d60dc0, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.148969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.168096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1087/2138] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:51 2022] PUT /identity/v3/projects/4d0078939cee4ad3a68c82e37a4c68f1/users/ab574d9ebed3485587512b2f82d60dc0/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:51.178576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.178775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.179105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] MFA Rules not processed for user `ab574d9ebed3485587512b2f82d60dc0`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:51.231472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:51.318123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1052/2139] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:51 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:51.322822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.408856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.410315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:51.533120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad25c6a5-3a98-4954-b87d-b3f6d01479ba None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:51.543543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47077b9-72ec-44e9-81d9-91fb79163d8f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:51.560291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.570903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.571028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.571028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.571028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.571028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.571418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.571503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.571503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] PATH_INFO: `/v3/projects/9e1006c0d7b44dc58cbb7786d7a78213` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.571503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e1006c0d7b44dc58cbb7786d7a78213)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.571739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.573590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.642418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1088/2140] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:51 2022] POST /identity/v3/projects => generated 375 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:51.643421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.693284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e1006c0d7b44dc58cbb7786d7a78213', 'request_id': 'req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:51.698764 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e1006c0d7b44dc58cbb7786d7a78213', 'request_id': 'req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:51.709088 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e1006c0d7b44dc58cbb7786d7a78213', 'request_id': 'req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:51.732494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] The token cache is being invalidate because project 9e1006c0d7b44dc58cbb7786d7a78213 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:51.732767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:51.733518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:51.738510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1053/2141] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:51 2022] DELETE /identity/v3/projects/9e1006c0d7b44dc58cbb7786d7a78213 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:51.739644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.741224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:51.762271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:51.778672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f5be87-d4ac-4945-8130-ca3099b9e2ba admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:51.793750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:51.861049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e39d12-0a9e-4693-83c5-bb1d72fcd6e3 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:51.865267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.865967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.866253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.866593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.867458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.875292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.876293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.876591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.876877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] PATH_INFO: `/v3/users/6bec97e7cee94e72ab43370f07088d02` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.878347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bec97e7cee94e72ab43370f07088d02)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.883369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.912483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1089/2142] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:51 2022] POST /identity/v3/users => generated 294 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:51.917737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.920364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:51.951764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d9df6-f4c6-498d-a908-eec3a824296a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:51.957708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:51.958111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:51.958899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:51.958899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:51.959642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:51.961288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:51.975897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bec97e7cee94e72ab43370f07088d02'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:51.981244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1090/2143] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:41:51.985369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:51.989576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bec97e7cee94e72ab43370f07088d02'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.003624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bec97e7cee94e72ab43370f07088d02'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.004620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bec97e7cee94e72ab43370f07088d02'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.013431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d5dc6c-b9ab-4643-9ef0-cc9f48b3c261 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:52.025381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:52.026776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:52.026776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:52.027543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] PATH_INFO: `/v3/projects/16e3ff5e1ba840f78b328f402edd6aa0/users/e7553b1856864cc497572357ef5534bc/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:52.033270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bec97e7cee94e72ab43370f07088d02'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.047483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16e3ff5e1ba840f78b328f402edd6aa0, user_id=e7553b1856864cc497572357ef5534bc, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:52.048934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1054/2144] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:51 2022] DELETE /identity/v3/users/6bec97e7cee94e72ab43370f07088d02 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:52.050821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:52.071191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1091/2145] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:51 2022] PUT /identity/v3/projects/16e3ff5e1ba840f78b328f402edd6aa0/users/e7553b1856864cc497572357ef5534bc/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:52.075456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:52.078202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:52.197568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ecbac-f912-4cd5-9ca2-e08ec7aaeaa1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:52.253680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:52.288590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae14fde-9919-4d7e-9b58-7b0d7a3341fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:52.289511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae14fde-9919-4d7e-9b58-7b0d7a3341fb admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:52.290002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae14fde-9919-4d7e-9b58-7b0d7a3341fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:52.290570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae14fde-9919-4d7e-9b58-7b0d7a3341fb admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:52.292562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae14fde-9919-4d7e-9b58-7b0d7a3341fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:52.294270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae14fde-9919-4d7e-9b58-7b0d7a3341fb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:52.312869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1055/2146] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:52.314277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:52.325596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e156b472-7c81-4421-b3e4-dcb70d1ae150 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:52.343391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:52.343618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:52.344566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:52.344690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] PATH_INFO: `/v3/projects/16e3ff5e1ba840f78b328f402edd6aa0/users/e7553b1856864cc497572357ef5534bc/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:52.346822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=16e3ff5e1ba840f78b328f402edd6aa0, user_id=e7553b1856864cc497572357ef5534bc, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:52.349827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:52.372560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1092/2147] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:52 2022] PUT /identity/v3/projects/16e3ff5e1ba840f78b328f402edd6aa0/users/e7553b1856864cc497572357ef5534bc/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:52.374880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:52.375252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:52.375611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:52.417347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] MFA Rules not processed for user `e7553b1856864cc497572357ef5534bc`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:52.418806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:52.511299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:52.514062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:52.579043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1056/2148] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:52 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:52.600087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:52.603127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:52.729416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad24f9d-bd2f-4c35-9950-aecc9e7d21fb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:52.740377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:52.741773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:52.742157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:52.742591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] PATH_INFO: `/v3/projects/764ebd462a2b4392b9b0c7ce86c5456e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:52.744758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=764ebd462a2b4392b9b0c7ce86c5456e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:52.747269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:52.760941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9acc056-116e-45bc-9add-7cb903427672 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:52.773324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95deb6b7-48aa-4da9-8210-b4929bd5eea5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:52.774996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95deb6b7-48aa-4da9-8210-b4929bd5eea5 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:52.775299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95deb6b7-48aa-4da9-8210-b4929bd5eea5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:52.775572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95deb6b7-48aa-4da9-8210-b4929bd5eea5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:52.781574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-95deb6b7-48aa-4da9-8210-b4929bd5eea5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:52.847363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '764ebd462a2b4392b9b0c7ce86c5456e', 'request_id': 'req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.862648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '764ebd462a2b4392b9b0c7ce86c5456e', 'request_id': 'req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.870664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:INSERT=6,keystone:DELETE=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:52.883085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '764ebd462a2b4392b9b0c7ce86c5456e', 'request_id': 'req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.912371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95deb6b7-48aa-4da9-8210-b4929bd5eea5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:52.913023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] The token cache is being invalidate because project 764ebd462a2b4392b9b0c7ce86c5456e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:52.913241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.915376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95deb6b7-48aa-4da9-8210-b4929bd5eea5 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:52.915737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-53b3f9ff-2464-4d3c-93dc-5c06318a8f2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:52.919975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1093/2149] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:52 2022] DELETE /identity/v3/projects/764ebd462a2b4392b9b0c7ce86c5456e => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:41:52.929826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd605bc-9257-4ec1-940f-b07a1c2e8920 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:52.930269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd605bc-9257-4ec1-940f-b07a1c2e8920 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:52.931395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd605bc-9257-4ec1-940f-b07a1c2e8920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:52.932844 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1057/2150] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:52 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:52.982556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4fd605bc-9257-4ec1-940f-b07a1c2e8920 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:52.984436 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4fd605bc-9257-4ec1-940f-b07a1c2e8920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:53.092637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1094/2151] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:53.100628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15cc953-0b28-4272-8a75-e075b46207f3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.101105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15cc953-0b28-4272-8a75-e075b46207f3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.101223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15cc953-0b28-4272-8a75-e075b46207f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.136542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c15cc953-0b28-4272-8a75-e075b46207f3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:53.137982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c15cc953-0b28-4272-8a75-e075b46207f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:53.154959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1058/2152] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:53.156492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd605bc-9257-4ec1-940f-b07a1c2e8920 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:53.177210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd605bc-9257-4ec1-940f-b07a1c2e8920 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:53.183461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:53.184263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.184556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.184837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.185595 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:53.185924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:53.188974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:53.210092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1095/2153] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:53.211414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15cc953-0b28-4272-8a75-e075b46207f3 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:53.222867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15cc953-0b28-4272-8a75-e075b46207f3 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:53.229306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:53.230081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.230464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.230684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.231503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:53.234262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:53.239207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:53.298941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1059/2154] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:53 2022] POST /identity/v3/projects => generated 375 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:53.303123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:53.322585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5996fa58-6892-4aaf-8dd0-3e59a6d0b5ce admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:53.433056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:53.433769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.434057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.434335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.435212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:53.437157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:53.474968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1096/2155] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:53 2022] POST /identity/v3/users => generated 288 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:53.480357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:53.510685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea2f23d-3b4d-4244-be4f-264d3989b1f3 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:53.525067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:53.526157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.526452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.527148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.527883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:53.533239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:53.562640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1060/2156] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:53 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:53.567742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:53.589998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65a5d7-a37d-4cb2-bfa5-16af213baab5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:53.630694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c338937-1279-46e9-a4d2-9c87bb90e203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:53.634582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c338937-1279-46e9-a4d2-9c87bb90e203 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.636066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c338937-1279-46e9-a4d2-9c87bb90e203 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.637469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c338937-1279-46e9-a4d2-9c87bb90e203 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.642979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c338937-1279-46e9-a4d2-9c87bb90e203 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:53.654088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c338937-1279-46e9-a4d2-9c87bb90e203 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:53.710045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1097/2157] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:53 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:53.711367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:53.727773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84321a8c-cad4-4774-beeb-b4b1a0bbad9c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:53.740267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:53.741192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.741480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.741769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] PATH_INFO: `/v3/projects/919ecc5dfeda4b4287a00b59149fc525/users/c3c616522dd54279881cc031a6b588d7/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.778551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] RBAC: Authorizing `identity:create_grant(project_id=919ecc5dfeda4b4287a00b59149fc525, user_id=c3c616522dd54279881cc031a6b588d7, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:53.781349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:53.820504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1061/2158] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:53 2022] PUT /identity/v3/projects/919ecc5dfeda4b4287a00b59149fc525/users/c3c616522dd54279881cc031a6b588d7/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:53.822329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.822847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.822935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:53.886715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] MFA Rules not processed for user `c3c616522dd54279881cc031a6b588d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:53.888089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:53.889267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:53.890545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:53.972181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:53.988928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f23e8-8752-462e-9f8f-a9a88d85664b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:53.996077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:53.998869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:53.999341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:53.999786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] PATH_INFO: `/v3/users/6736db4b6ba0481c921e1107dd87faf6` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:54.001567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6736db4b6ba0481c921e1107dd87faf6)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:54.003952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:54.017529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1098/2159] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:53 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:54.024601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:54.029972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:54.101257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6736db4b6ba0481c921e1107dd87faf6'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:54.112644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6736db4b6ba0481c921e1107dd87faf6'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:54.118017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6fb80-3c6b-4b71-a58e-ad2c1bf1d47d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:54.122743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6736db4b6ba0481c921e1107dd87faf6'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:54.123838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:54.124186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6736db4b6ba0481c921e1107dd87faf6'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:54.125235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:54.125605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:54.125962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:54.138672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:54.140727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:54.141953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4f552381-5090-4bd6-8f4a-d70ca7978bc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6736db4b6ba0481c921e1107dd87faf6'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:54.153103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1062/2160] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:53 2022] DELETE /identity/v3/users/6736db4b6ba0481c921e1107dd87faf6 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:54.155063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:54.155511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:54.155799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:54.166224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1099/2161] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:54 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:54.185890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:54.187178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:54.238192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:54.321966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1063/2162] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:54.333763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:54.335768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:54.477885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae16920a-c9ca-4ed1-b194-b96986157caa service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:54.692655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:54.693333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:54.693622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:54.693903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:54.733040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:54.740382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:54.782601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1100/2163] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:55.084599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:55.089677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:55.237147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dccfffc-787b-44df-8f54-2b7728f56800 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:55.242692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563ff9ca-2c2b-4ec4-8f01-ee3a7cfcdff8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:55.244578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563ff9ca-2c2b-4ec4-8f01-ee3a7cfcdff8 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:55.244873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563ff9ca-2c2b-4ec4-8f01-ee3a7cfcdff8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:55.245411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563ff9ca-2c2b-4ec4-8f01-ee3a7cfcdff8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:55.247580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-563ff9ca-2c2b-4ec4-8f01-ee3a7cfcdff8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:55.382808 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:55.383459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563ff9ca-2c2b-4ec4-8f01-ee3a7cfcdff8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:55.385277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:55.388237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563ff9ca-2c2b-4ec4-8f01-ee3a7cfcdff8 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:55.411288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1064/2164] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:55 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:55.428331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abf8762-8da4-4c01-8b7a-2e637a341688 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:55.444423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:55.445330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:55.445822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:55.446147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] PATH_INFO: `/v3/projects/52bd3580c4c342ebae801d2b710ac508` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:55.450015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52bd3580c4c342ebae801d2b710ac508)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:55.452037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:55.593372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:55.613295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52bd3580c4c342ebae801d2b710ac508', 'request_id': 'req-16be0ce7-4785-47a1-b310-981b2017d366'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:55.632986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52bd3580c4c342ebae801d2b710ac508', 'request_id': 'req-16be0ce7-4785-47a1-b310-981b2017d366'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:55.660071 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52bd3580c4c342ebae801d2b710ac508', 'request_id': 'req-16be0ce7-4785-47a1-b310-981b2017d366'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:55.689906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] The token cache is being invalidate because project 52bd3580c4c342ebae801d2b710ac508 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:41:55.690575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:55.691806 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-16be0ce7-4785-47a1-b310-981b2017d366 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:55.697614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1101/2165] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:41:55 2022] DELETE /identity/v3/projects/52bd3580c4c342ebae801d2b710ac508 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:55.699471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fe3f74-fa00-4b0a-a323-c695808a45b1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:55.699471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fe3f74-fa00-4b0a-a323-c695808a45b1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:55.700343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fe3f74-fa00-4b0a-a323-c695808a45b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:55.772580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f0fe3f74-fa00-4b0a-a323-c695808a45b1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:55.773833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f0fe3f74-fa00-4b0a-a323-c695808a45b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:55.957372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1065/2166] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:55.967233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e83ce-4435-4f88-ab3f-d55b53343e75 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:55.967535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e83ce-4435-4f88-ab3f-d55b53343e75 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:55.967822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e83ce-4435-4f88-ab3f-d55b53343e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.004100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-945e83ce-4435-4f88-ab3f-d55b53343e75 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:56.005410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-945e83ce-4435-4f88-ab3f-d55b53343e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:56.031830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1102/2167] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:56.035992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fe3f74-fa00-4b0a-a323-c695808a45b1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:56.052928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fe3f74-fa00-4b0a-a323-c695808a45b1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:56.067366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:56.069674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.070007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.070296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.073269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:41:56.073596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:56.077324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:56.109789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1066/2168] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:41:56.112538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945e83ce-4435-4f88-ab3f-d55b53343e75 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:56.137315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945e83ce-4435-4f88-ab3f-d55b53343e75 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:56.143846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:56.144983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.145536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.146110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.148043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:56.156344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:56.204372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1103/2169] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:41:56 2022] POST /identity/v3/projects => generated 383 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:41:56.209937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:56.225885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca150d2-4e6e-4901-b805-b3e4b6915c8a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:56.257559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:56.433519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:56.434197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.434514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.434802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.435532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:56.447617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:56.489707 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1067/2170] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:41:56 2022] POST /identity/v3/users => generated 292 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:41:56.490869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:56.511189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87487133-d1eb-4587-a07d-476299db57fb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:56.540585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:56.541987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.542958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.544182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.547540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:56.556374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:56.592519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1104/2171] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:56 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:56.593668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:56.621559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43def438-0857-4fac-80a7-e0ce2e40b523 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:56.643004 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:56.643702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.643993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.644278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.645028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:56.646969 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:56.667791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1068/2172] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:41:56 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:41:56.671706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:56.694553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:56.696731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:56.719709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da82bd8-9ce2-455d-975e-fd9505d0e7de admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:56.727689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c4bbb9-4fcb-4df4-aa2c-34387ac82155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:56.729135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4bbb9-4fcb-4df4-aa2c-34387ac82155 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.729912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4bbb9-4fcb-4df4-aa2c-34387ac82155 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.730554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4bbb9-4fcb-4df4-aa2c-34387ac82155 admin admin] PATH_INFO: `/v3/projects/8e0ccc1e4683478889d90814a1305507/users/b7457a38a0594e8da4c053a5c6a4aa15/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.754960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c4bbb9-4fcb-4df4-aa2c-34387ac82155 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e0ccc1e4683478889d90814a1305507, user_id=b7457a38a0594e8da4c053a5c6a4aa15, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:56.760794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c4bbb9-4fcb-4df4-aa2c-34387ac82155 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:56.767214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a4201a-5fd9-4c96-b88c-f3db4da3fc48 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:56.771493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:56.772214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.773515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.773515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.775162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:56.795198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1105/2173] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:41:56 2022] PUT /identity/v3/projects/8e0ccc1e4683478889d90814a1305507/users/b7457a38a0594e8da4c053a5c6a4aa15/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:56.801043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5625c608-20e2-473a-9db0-8653d34e53e8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:56.801341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5625c608-20e2-473a-9db0-8653d34e53e8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:56.801624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5625c608-20e2-473a-9db0-8653d34e53e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:56.847912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5625c608-20e2-473a-9db0-8653d34e53e8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:56.864853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5625c608-20e2-473a-9db0-8653d34e53e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:56.998262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:57.001610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:57.043713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1069/2174] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:56 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:57.063900 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1106/2175] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:57.071018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:57.072947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:57.186206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:57.222894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be84e370-68d3-4475-9145-8f750eb16109 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:57.305908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:57.306688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:57.307043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:57.307509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:57.328673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:57.331623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:57.351639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1070/2176] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:57.971755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:57.972653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:57.973720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:58.011348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] MFA Rules not processed for user `d652ee5c440b41c980191f5c0c1eeb8e`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:58.012915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:58.086825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:58.119854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1107/2177] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:57 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:58.136677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:58.138130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:58.223663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e9fd7d-8f16-4457-b608-e40044f1ceb3 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:58.232306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b76790-97f2-4750-889a-9525b32f5ca2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:58.233358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b76790-97f2-4750-889a-9525b32f5ca2 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:58.233838 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b76790-97f2-4750-889a-9525b32f5ca2 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:58.234974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b76790-97f2-4750-889a-9525b32f5ca2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:58.237780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d2b76790-97f2-4750-889a-9525b32f5ca2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:58.311225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b76790-97f2-4750-889a-9525b32f5ca2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:58.313194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b76790-97f2-4750-889a-9525b32f5ca2 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:58.325501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1071/2178] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:58 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:41:58.610747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:58.612250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:58.637008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3ab76-282b-43d1-a2c3-448c41c9436b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:58.646206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:58.647726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:58.648360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:58.649313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] PATH_INFO: `/v3/users/c3c616522dd54279881cc031a6b588d7` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:58.650970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3c616522dd54279881cc031a6b588d7)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:58.653510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:58.719807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:58.720132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:58.720422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:58.733761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3c616522dd54279881cc031a6b588d7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:58.747968 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3c616522dd54279881cc031a6b588d7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:58.748767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3c616522dd54279881cc031a6b588d7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:58.772213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3c616522dd54279881cc031a6b588d7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:58.805250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] MFA Rules not processed for user `b7457a38a0594e8da4c053a5c6a4aa15`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:58.806660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:58.818717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2a8aab6e-d325-4c2d-8f7e-9759db7ec7e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3c616522dd54279881cc031a6b588d7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:41:58.835480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1108/2179] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:41:58 2022] DELETE /identity/v3/users/c3c616522dd54279881cc031a6b588d7 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:41:58.857389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:58.857701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:58.857974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:58.874525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:41:58.910062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:41:58.913066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:58.977400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1072/2180] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:41:58.998908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:59.000875 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:59.054809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1109/2181] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:41:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:41:59.073456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:41:59.075142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:41:59.146212 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a23047-dbd9-4f39-883e-961e75e4b4c1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:59.155331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4149d3-531a-4e77-ae1c-0e8fa23d16bc None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:41:59.159570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:59.160348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:59.160638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:59.160916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:59.174655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:59.176059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:59.194768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1110/2182] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:41:59.242059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b50c07-7451-4575-ba6e-24b62fc56097 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:41:59.243423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b50c07-7451-4575-ba6e-24b62fc56097 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:41:59.244954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b50c07-7451-4575-ba6e-24b62fc56097 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:41:59.245253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b50c07-7451-4575-ba6e-24b62fc56097 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:41:59.327937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b50c07-7451-4575-ba6e-24b62fc56097 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:41:59.330277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b50c07-7451-4575-ba6e-24b62fc56097 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:41:59.343332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1073/2183] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:41:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:42:00.038421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:00.041011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:00.053476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:00.064323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3d9987-3e67-4bb7-8dde-fe1ec9ca36cf service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:00.072899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:00.073943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:00.074336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:00.074749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] PATH_INFO: `/v3/projects/919ecc5dfeda4b4287a00b59149fc525` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:00.076157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=919ecc5dfeda4b4287a00b59149fc525)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:00.079308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:00.206338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '919ecc5dfeda4b4287a00b59149fc525', 'request_id': 'req-12503277-6df0-44b1-9146-f4407dcddd5a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:00.217021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '919ecc5dfeda4b4287a00b59149fc525', 'request_id': 'req-12503277-6df0-44b1-9146-f4407dcddd5a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:00.227211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '919ecc5dfeda4b4287a00b59149fc525', 'request_id': 'req-12503277-6df0-44b1-9146-f4407dcddd5a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:00.247178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] The token cache is being invalidate because project 919ecc5dfeda4b4287a00b59149fc525 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:42:00.247945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:00.249216 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:00.250840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1111/2184] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:42:00 2022] DELETE /identity/v3/projects/919ecc5dfeda4b4287a00b59149fc525 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:00.970086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f2e0ce-96da-457f-b545-dc7f669cdaff None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:00.970608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f2e0ce-96da-457f-b545-dc7f669cdaff None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:00.970730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f2e0ce-96da-457f-b545-dc7f669cdaff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:00.987548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-54f2e0ce-96da-457f-b545-dc7f669cdaff None None] MFA Rules not processed for user `b7457a38a0594e8da4c053a5c6a4aa15`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:00.988757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-54f2e0ce-96da-457f-b545-dc7f669cdaff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:01.020237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:01.047737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1074/2185] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:01.378386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:01.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:01.460826 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12503277-6df0-44b1-9146-f4407dcddd5a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:01.466038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:01.467038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:01.467330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:01.467677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] PATH_INFO: `/v3/users/eac3714a97064436be7501249aba35eb` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:01.471816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eac3714a97064436be7501249aba35eb)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:01.473964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:01.532994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eac3714a97064436be7501249aba35eb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:01.546124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eac3714a97064436be7501249aba35eb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:01.546803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eac3714a97064436be7501249aba35eb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:01.556747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eac3714a97064436be7501249aba35eb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:01.583762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eac3714a97064436be7501249aba35eb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:01.591101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1112/2186] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:42:01 2022] DELETE /identity/v3/users/eac3714a97064436be7501249aba35eb => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:01.595993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:01.596162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:01.596322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:01.625684 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:01.626787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:01.747536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1075/2187] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:01.758886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:01.760860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:01.769690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:01.846705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d938bf37-81c7-499e-b7a8-c6c7a48477e6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:01.851236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:01.852413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:01.852863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:01.853297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:01.872308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:01.874535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:01.888860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1113/2188] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:42:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:42:02.227097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:02.229176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:02.242710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2192a2-6e92-48aa-9a62-63ffecc6ac6d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:02.251612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:02.252698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:02.253301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:02.253901 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] PATH_INFO: `/v3/projects/e33331b4e057438a9aa0bbcffc23f089` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:02.257935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e33331b4e057438a9aa0bbcffc23f089)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:02.259924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:02.331779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e33331b4e057438a9aa0bbcffc23f089', 'request_id': 'req-25119ade-c2df-4e42-9ed8-cb9d89d2c640'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.344216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e33331b4e057438a9aa0bbcffc23f089', 'request_id': 'req-25119ade-c2df-4e42-9ed8-cb9d89d2c640'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.357945 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e33331b4e057438a9aa0bbcffc23f089', 'request_id': 'req-25119ade-c2df-4e42-9ed8-cb9d89d2c640'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.372440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] The token cache is being invalidate because project e33331b4e057438a9aa0bbcffc23f089 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:42:02.372736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.373470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.375315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1076/2189] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:42:02 2022] DELETE /identity/v3/projects/e33331b4e057438a9aa0bbcffc23f089 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:02.379425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:02.391991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beb4a88-b2e3-45df-8dca-06638143f67d service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:02.466380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:02.467605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:02.467907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:02.468164 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] PATH_INFO: `/v3/users/ab574d9ebed3485587512b2f82d60dc0` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:02.469576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab574d9ebed3485587512b2f82d60dc0)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:02.471622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:02.526464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab574d9ebed3485587512b2f82d60dc0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.534337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab574d9ebed3485587512b2f82d60dc0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.534994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab574d9ebed3485587512b2f82d60dc0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.544789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab574d9ebed3485587512b2f82d60dc0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.558771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab574d9ebed3485587512b2f82d60dc0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:02.565162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1114/2190] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:42:02 2022] DELETE /identity/v3/users/ab574d9ebed3485587512b2f82d60dc0 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:42:02.941247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:02.942891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:02.967853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25119ade-c2df-4e42-9ed8-cb9d89d2c640 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:03.045906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:03.047283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:03.047925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:03.055127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] PATH_INFO: `/v3/projects/4d0078939cee4ad3a68c82e37a4c68f1` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:03.057002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d0078939cee4ad3a68c82e37a4c68f1)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:03.059369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:03.066762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:03.124215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d0078939cee4ad3a68c82e37a4c68f1', 'request_id': 'req-11138948-b3d2-439e-a2cc-f31bc7b1ea63'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.130855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d0078939cee4ad3a68c82e37a4c68f1', 'request_id': 'req-11138948-b3d2-439e-a2cc-f31bc7b1ea63'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.143930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d0078939cee4ad3a68c82e37a4c68f1', 'request_id': 'req-11138948-b3d2-439e-a2cc-f31bc7b1ea63'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.162271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] The token cache is being invalidate because project 4d0078939cee4ad3a68c82e37a4c68f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:42:03.162596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.163259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.165868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1077/2191] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:42:02 2022] DELETE /identity/v3/projects/4d0078939cee4ad3a68c82e37a4c68f1 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:03.169971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:03.171935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:03.188099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5096e03b-79e9-4c9a-86f5-803dae0c3693 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:03.230407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:03.291941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:03.293201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:03.293530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:03.293914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] PATH_INFO: `/v3/users/e7553b1856864cc497572357ef5534bc` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:03.295356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7553b1856864cc497572357ef5534bc)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:03.300002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:03.388384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7553b1856864cc497572357ef5534bc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.399603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7553b1856864cc497572357ef5534bc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.400442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7553b1856864cc497572357ef5534bc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.408719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7553b1856864cc497572357ef5534bc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.424023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-b7942711-2f34-48de-b9c4-f48c4149300a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7553b1856864cc497572357ef5534bc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:03.432405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1115/2192] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:42:03 2022] DELETE /identity/v3/users/e7553b1856864cc497572357ef5534bc => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:42:04.519711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:04.521279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:04.536105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11138948-b3d2-439e-a2cc-f31bc7b1ea63 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:04.645553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:04.647014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:04.647451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:04.647904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] PATH_INFO: `/v3/projects/16e3ff5e1ba840f78b328f402edd6aa0` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:04.650174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] RBAC: Authorizing `identity:delete_project(project_id=16e3ff5e1ba840f78b328f402edd6aa0)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:04.652425 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:04.720072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16e3ff5e1ba840f78b328f402edd6aa0', 'request_id': 'req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:04.731929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16e3ff5e1ba840f78b328f402edd6aa0', 'request_id': 'req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:04.741585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16e3ff5e1ba840f78b328f402edd6aa0', 'request_id': 'req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:04.757023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] The token cache is being invalidate because project 16e3ff5e1ba840f78b328f402edd6aa0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:42:04.757318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:04.758087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fda8b7d2-f7c9-460e-9ead-ca9d4560985e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:04.760423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1078/2193] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:42:04 2022] DELETE /identity/v3/projects/16e3ff5e1ba840f78b328f402edd6aa0 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:04.765687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883282a4-b16f-4672-9372-4840dc5ea117 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:04.766179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883282a4-b16f-4672-9372-4840dc5ea117 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:04.766458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883282a4-b16f-4672-9372-4840dc5ea117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:04.785467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-883282a4-b16f-4672-9372-4840dc5ea117 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:04.786474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-883282a4-b16f-4672-9372-4840dc5ea117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:04.855563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1116/2194] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:04.859609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b26a95-187a-4a1b-b320-ead877efde5f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:04.859968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b26a95-187a-4a1b-b320-ead877efde5f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:04.860271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b26a95-187a-4a1b-b320-ead877efde5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:04.888105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-37b26a95-187a-4a1b-b320-ead877efde5f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:04.889347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-37b26a95-187a-4a1b-b320-ead877efde5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:04.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1079/2195] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:04.903250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883282a4-b16f-4672-9372-4840dc5ea117 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:04.914073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883282a4-b16f-4672-9372-4840dc5ea117 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:04.919824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:04.920634 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:04.920993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:04.921346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:04.922162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:42:04.922673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:04.924281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:04.938098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1117/2196] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:42:04.944022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b26a95-187a-4a1b-b320-ead877efde5f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:04.956754 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b26a95-187a-4a1b-b320-ead877efde5f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:04.962629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:04.963311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:04.963600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:04.963878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:04.964602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:04.966753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:04.992318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:05.011225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1080/2197] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:42:04 2022] POST /identity/v3/projects => generated 363 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:42:05.012094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:05.024709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e17d328-e62e-426b-b49a-0cd88cca09e1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:05.072731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:05.121474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:05.121912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:05.122106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:05.122465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:05.123290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:05.126534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:05.150466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1118/2198] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:42:05 2022] POST /identity/v3/users => generated 282 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:42:05.153242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:05.163970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c1d96e-6305-43e0-beb6-83b79e79ab1b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:05.169270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:05.169951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:05.170240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:05.170559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:05.171300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:05.173079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:05.184711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1081/2199] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:42:05.189256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:05.199564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821bc2a4-9328-4ca9-8286-04db992e665e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:05.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32753ada-bdfc-418e-848c-b6fc9fc42e0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:05.205207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32753ada-bdfc-418e-848c-b6fc9fc42e0d admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:05.205750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32753ada-bdfc-418e-848c-b6fc9fc42e0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:05.206028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32753ada-bdfc-418e-848c-b6fc9fc42e0d admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:05.206789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32753ada-bdfc-418e-848c-b6fc9fc42e0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:05.208523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32753ada-bdfc-418e-848c-b6fc9fc42e0d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:05.219663 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1119/2200] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:42:05.223822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:05.235234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c76dc8-1b78-4d13-b30a-1ec73521912d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:05.239849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:05.240741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:05.241017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:05.241290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] PATH_INFO: `/v3/projects/143948d3b49640d0a5a0b3375448f576/users/2bf0eb70df604c528051d56ef5e130e3/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:05.257090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=143948d3b49640d0a5a0b3375448f576, user_id=2bf0eb70df604c528051d56ef5e130e3, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:05.260401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:05.274201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1082/2201] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:42:05 2022] PUT /identity/v3/projects/143948d3b49640d0a5a0b3375448f576/users/2bf0eb70df604c528051d56ef5e130e3/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:05.285181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a33b48-f619-4153-ac07-c8faf5f6f79d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:05.285501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a33b48-f619-4153-ac07-c8faf5f6f79d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:05.285767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a33b48-f619-4153-ac07-c8faf5f6f79d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:05.315701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-00a33b48-f619-4153-ac07-c8faf5f6f79d None None] MFA Rules not processed for user `2bf0eb70df604c528051d56ef5e130e3`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:05.316954 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-00a33b48-f619-4153-ac07-c8faf5f6f79d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:05.392374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1120/2202] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:05 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:05.404339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:05.406098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:05.474481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb243c0-4290-4a17-a087-85d3d7fe0c1e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:05.478925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f4acd5-512c-4412-a90b-1cad137cf618 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:05.479639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4acd5-512c-4412-a90b-1cad137cf618 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:05.480572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4acd5-512c-4412-a90b-1cad137cf618 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:05.480572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4acd5-512c-4412-a90b-1cad137cf618 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:05.490804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4acd5-512c-4412-a90b-1cad137cf618 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:05.492350 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4acd5-512c-4412-a90b-1cad137cf618 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:05.506688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1083/2203] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:42:05 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:42:15.388926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=51,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:15.498272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=43,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:21.201706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:21.202285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:21.202285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:21.234099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] MFA Rules not processed for user `3ae56fd1172f4bf8806eefc6e8877fef`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:21.235371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:21.292137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1121/2204] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:25.251939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17e4a9-92f8-4772-b33a-397934012840 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:25.252587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17e4a9-92f8-4772-b33a-397934012840 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:25.252587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17e4a9-92f8-4772-b33a-397934012840 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:25.275122 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bf17e4a9-92f8-4772-b33a-397934012840 None None] MFA Rules not processed for user `0f8e300cd96e446db7430044e69d997a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:25.276110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bf17e4a9-92f8-4772-b33a-397934012840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:25.342897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1084/2205] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:25 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:31.285272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=25 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:35.335584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=25 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:44.333353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:44.334857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:44.393747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44fc1e7-b954-4f44-b4ce-cca1cd5939ad None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:44.397508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835eca85-7fa2-41c9-8ea9-66b4b9913ddf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:44.398179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835eca85-7fa2-41c9-8ea9-66b4b9913ddf service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:44.398995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835eca85-7fa2-41c9-8ea9-66b4b9913ddf service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:44.399257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835eca85-7fa2-41c9-8ea9-66b4b9913ddf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:44.401068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-835eca85-7fa2-41c9-8ea9-66b4b9913ddf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:44.468952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835eca85-7fa2-41c9-8ea9-66b4b9913ddf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:44.470676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835eca85-7fa2-41c9-8ea9-66b4b9913ddf service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:44.484577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1122/2206] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:42:44 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:42:50.789527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf17e4a9-92f8-4772-b33a-397934012840 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:50.857706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf17e4a9-92f8-4772-b33a-397934012840 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:50.862143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:50.863056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:50.863340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:50.863616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] PATH_INFO: `/v3/users/2bf0eb70df604c528051d56ef5e130e3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:50.864791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bf0eb70df604c528051d56ef5e130e3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:50.866772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:50.916024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bf0eb70df604c528051d56ef5e130e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:50.923365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bf0eb70df604c528051d56ef5e130e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:50.931259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bf0eb70df604c528051d56ef5e130e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:50.931845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bf0eb70df604c528051d56ef5e130e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:50.943910 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bf0eb70df604c528051d56ef5e130e3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:50.951242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1085/2207] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:42:50 2022] DELETE /identity/v3/users/2bf0eb70df604c528051d56ef5e130e3 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:50.955524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:50.956398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:50.956398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:50.975417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:50.977568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:51.052241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1123/2208] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:51.062191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:51.063832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:51.129619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6aaa29-dabf-4c9a-a11a-4c90a03f570c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:51.133782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee46c19-ef9c-4707-8ffb-53c682d1b311 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:51.134458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee46c19-ef9c-4707-8ffb-53c682d1b311 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.134748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee46c19-ef9c-4707-8ffb-53c682d1b311 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.135026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee46c19-ef9c-4707-8ffb-53c682d1b311 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.145811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee46c19-ef9c-4707-8ffb-53c682d1b311 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:51.147897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee46c19-ef9c-4707-8ffb-53c682d1b311 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:51.158384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1086/2209] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:42:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:42:51.451353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:51.453043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:51.468428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1519b264-3630-4a51-ae31-9fe676946312 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:51.473823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:51.475108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.475398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.475676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] PATH_INFO: `/v3/projects/143948d3b49640d0a5a0b3375448f576` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.477445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=143948d3b49640d0a5a0b3375448f576)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:51.480890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:51.529100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:51.533103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '143948d3b49640d0a5a0b3375448f576', 'request_id': 'req-47b37996-686b-4c54-9e6d-02e4fce293ac'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:51.541298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '143948d3b49640d0a5a0b3375448f576', 'request_id': 'req-47b37996-686b-4c54-9e6d-02e4fce293ac'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:51.547297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '143948d3b49640d0a5a0b3375448f576', 'request_id': 'req-47b37996-686b-4c54-9e6d-02e4fce293ac'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:51.559713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] The token cache is being invalidate because project 143948d3b49640d0a5a0b3375448f576 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:42:51.560006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:51.560670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-47b37996-686b-4c54-9e6d-02e4fce293ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:51.562225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1124/2210] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:42:51 2022] DELETE /identity/v3/projects/143948d3b49640d0a5a0b3375448f576 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:51.567173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9649d0-acd3-4c82-801e-04f4ab5be6be None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.567472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9649d0-acd3-4c82-801e-04f4ab5be6be None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.567759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9649d0-acd3-4c82-801e-04f4ab5be6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.585555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ed9649d0-acd3-4c82-801e-04f4ab5be6be None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:51.586790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ed9649d0-acd3-4c82-801e-04f4ab5be6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:51.603974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:51.647418 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1087/2211] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:51.655074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3914d2-800b-44c2-af18-b36c67323f2e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.655253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3914d2-800b-44c2-af18-b36c67323f2e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.655538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3914d2-800b-44c2-af18-b36c67323f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.677695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4c3914d2-800b-44c2-af18-b36c67323f2e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:51.678961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4c3914d2-800b-44c2-af18-b36c67323f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:51.691324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1125/2212] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:51.695288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9649d0-acd3-4c82-801e-04f4ab5be6be None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:51.712186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9649d0-acd3-4c82-801e-04f4ab5be6be None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:51.718416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:51.719181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.719400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.720369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.721178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:42:51.721493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:51.723770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:51.740010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1088/2213] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:42:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:42:51.744596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3914d2-800b-44c2-af18-b36c67323f2e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:51.755739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3914d2-800b-44c2-af18-b36c67323f2e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:51.760489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:51.761151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.761431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.761707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.762419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:51.764366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:51.808284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1126/2214] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:42:51 2022] POST /identity/v3/projects => generated 373 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:42:51.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:51.822912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77adea34-24b3-4715-b4b2-299189072b43 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:51.884409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:51.885317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.885674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.886034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.886809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:51.888749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:51.910470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1089/2215] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:42:51 2022] POST /identity/v3/users => generated 287 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:42:51.914619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:51.926365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572fd188-8a67-4e6a-8d31-94cb50a0beb4 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:51.933587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:51.933587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.933974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.934083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.935289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:51.936936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:51.950725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1127/2216] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:42:51 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:42:51.954669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:51.966398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b02bd6f-f635-4170-9f3c-75f396b8da71 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:51.971611 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2b1255-0227-49a8-8a52-9cdc55ca94fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:51.972303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b1255-0227-49a8-8a52-9cdc55ca94fe admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:51.972605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b1255-0227-49a8-8a52-9cdc55ca94fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:51.972967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b1255-0227-49a8-8a52-9cdc55ca94fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:51.973639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2b1255-0227-49a8-8a52-9cdc55ca94fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:51.975666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2b1255-0227-49a8-8a52-9cdc55ca94fe admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:51.987323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1090/2217] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:42:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:42:51.991587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:52.002794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b435b544-2ae6-4bda-bd96-604a4988df8d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:52.008284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:52.009393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:52.009840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:52.010121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] PATH_INFO: `/v3/projects/652f8c451cba457d92d8519659ef9406/users/d1e34064344443faad3ce80b7440406e/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:52.025923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] RBAC: Authorizing `identity:create_grant(project_id=652f8c451cba457d92d8519659ef9406, user_id=d1e34064344443faad3ce80b7440406e, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:52.028672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:52.042814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1128/2218] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:42:51 2022] PUT /identity/v3/projects/652f8c451cba457d92d8519659ef9406/users/d1e34064344443faad3ce80b7440406e/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:52.053701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f2938-7985-4dc9-97b1-b449f8806c74 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:52.054256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f2938-7985-4dc9-97b1-b449f8806c74 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:52.055102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f2938-7985-4dc9-97b1-b449f8806c74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:52.084958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7a9f2938-7985-4dc9-97b1-b449f8806c74 None None] MFA Rules not processed for user `d1e34064344443faad3ce80b7440406e`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:52.086158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7a9f2938-7985-4dc9-97b1-b449f8806c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:52.153642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1091/2219] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:52 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:42:52.165529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:52.167314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:52.271769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbccec-754e-43ae-b9b6-e92f24cee374 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:52.276294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f5be21-31e8-4bca-8e44-cdda0b7989a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:52.277083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f5be21-31e8-4bca-8e44-cdda0b7989a2 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:52.277441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f5be21-31e8-4bca-8e44-cdda0b7989a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:52.277847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f5be21-31e8-4bca-8e44-cdda0b7989a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:52.288285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f5be21-31e8-4bca-8e44-cdda0b7989a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:52.290924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f5be21-31e8-4bca-8e44-cdda0b7989a2 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:52.318604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1129/2220] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:42:52 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:42:52.343775 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f2938-7985-4dc9-97b1-b449f8806c74 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:52.368577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:52.429944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f2938-7985-4dc9-97b1-b449f8806c74 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:52.437902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:52.439205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:52.439630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:52.440009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] PATH_INFO: `/v3/users/d1e34064344443faad3ce80b7440406e` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:52.442032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1e34064344443faad3ce80b7440406e)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:52.444346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:52.503183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1e34064344443faad3ce80b7440406e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:52.510337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1e34064344443faad3ce80b7440406e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:52.518835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1e34064344443faad3ce80b7440406e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:52.519955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1e34064344443faad3ce80b7440406e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:52.533587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1e34064344443faad3ce80b7440406e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:52.541323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1092/2221] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:42:52 2022] DELETE /identity/v3/users/d1e34064344443faad3ce80b7440406e => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:52.545986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fc0d4e-e817-44f8-bec6-a3c98218f9d5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:52.546289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fc0d4e-e817-44f8-bec6-a3c98218f9d5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:52.546614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fc0d4e-e817-44f8-bec6-a3c98218f9d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:52.564830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c1fc0d4e-e817-44f8-bec6-a3c98218f9d5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:52.566109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c1fc0d4e-e817-44f8-bec6-a3c98218f9d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:52.626912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1130/2222] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:52.636575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:52.638066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:52.699888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f44569-8d6f-484e-b810-f3f7d9188be5 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:52.703538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:52.704191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:52.704852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:52.704852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:52.715196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:52.716644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:52.727380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1093/2223] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:42:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:42:52.922076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:52.922686 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:52.923107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:52.959914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:52.962069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:52.989867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:53.055659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1131/2224] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:42:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:53.121095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:53.122925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:53.181625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:53.211533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:53.213161 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:53.221233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b021efab-023d-4ee7-b712-59c99c135286 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:53.228543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c62dca-8f52-47e1-9e6b-353c14c20386 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:53.229679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c62dca-8f52-47e1-9e6b-353c14c20386 service swift] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.229679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c62dca-8f52-47e1-9e6b-353c14c20386 service swift] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.229883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c62dca-8f52-47e1-9e6b-353c14c20386 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.233067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399b8fe9-5600-443b-b0ec-5610d0b6ce5c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:53.238204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:53.239387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.239828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.239949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c62dca-8f52-47e1-9e6b-353c14c20386 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:53.240362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] PATH_INFO: `/v3/projects/652f8c451cba457d92d8519659ef9406` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.241631 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c62dca-8f52-47e1-9e6b-353c14c20386 service swift] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:53.243180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] RBAC: Authorizing `identity:delete_project(project_id=652f8c451cba457d92d8519659ef9406)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:53.246257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:53.252925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1094/2225] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:42:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:42:53.346143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '652f8c451cba457d92d8519659ef9406', 'request_id': 'req-d23db4c6-97a0-4d89-a0da-3c0a5c327838'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:53.360365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '652f8c451cba457d92d8519659ef9406', 'request_id': 'req-d23db4c6-97a0-4d89-a0da-3c0a5c327838'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:53.382578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '652f8c451cba457d92d8519659ef9406', 'request_id': 'req-d23db4c6-97a0-4d89-a0da-3c0a5c327838'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:53.413258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] The token cache is being invalidate because project 652f8c451cba457d92d8519659ef9406 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:42:53.413871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:53.417283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0bf5d5-f1b8-45d9-b111-1c0bb6a5ef68 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.417772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0bf5d5-f1b8-45d9-b111-1c0bb6a5ef68 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.417772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0bf5d5-f1b8-45d9-b111-1c0bb6a5ef68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.420542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d23db4c6-97a0-4d89-a0da-3c0a5c327838 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:42:53.423575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1132/2226] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:42:53 2022] DELETE /identity/v3/projects/652f8c451cba457d92d8519659ef9406 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:53.428289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cbc0bd-2497-4b7c-bd39-6d2ea89b6ebc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.429105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cbc0bd-2497-4b7c-bd39-6d2ea89b6ebc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.430129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cbc0bd-2497-4b7c-bd39-6d2ea89b6ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.456317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-21cbc0bd-2497-4b7c-bd39-6d2ea89b6ebc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:53.457805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-21cbc0bd-2497-4b7c-bd39-6d2ea89b6ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:53.471320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8a0bf5d5-f1b8-45d9-b111-1c0bb6a5ef68 None None] MFA Rules not processed for user `fa478fb4b41e49c5b1f9291f925cca23`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:53.486665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8a0bf5d5-f1b8-45d9-b111-1c0bb6a5ef68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:53.533837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1133/2227] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:42:53.542045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54446291-feff-4c16-8b4c-d2ead8b4340d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.542508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54446291-feff-4c16-8b4c-d2ead8b4340d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.543466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54446291-feff-4c16-8b4c-d2ead8b4340d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.595374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-54446291-feff-4c16-8b4c-d2ead8b4340d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:53.596877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-54446291-feff-4c16-8b4c-d2ead8b4340d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:53.609262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1095/2228] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:53.623433 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1134/2229] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:42:53.624641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0bf5d5-f1b8-45d9-b111-1c0bb6a5ef68 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:53.643866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0bf5d5-f1b8-45d9-b111-1c0bb6a5ef68 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:53.662474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:53.663179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.663470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.664343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.665090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:42:53.665407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:53.667127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:53.692601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1096/2230] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:42:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:42:53.693874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54446291-feff-4c16-8b4c-d2ead8b4340d None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:53.717323 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54446291-feff-4c16-8b4c-d2ead8b4340d None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:53.723414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:53.724256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.724651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.725040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.726792 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:53.728523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:53.788966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1135/2231] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:42:53 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:42:53.789472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:53.808130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208127e3-18ed-46b2-93f8-deed318b723d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:53.925493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:53.926413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.926822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.927209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.928060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:53.930202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:53.960080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1097/2232] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:42:53 2022] POST /identity/v3/users => generated 293 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:42:53.964261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:53.979761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef07b526-d48f-475d-9923-0e9a97b4d406 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:53.985288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:53.985966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:53.986256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:53.986576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:53.987321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:53.988945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:54.007573 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1136/2233] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:42:53 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:42:54.008559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:54.022481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92be1b3-35cc-4f1e-8bb8-bb1e2a6196a4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:54.027975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be366b4-7891-46e6-89e5-c8c79ac042e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:54.028750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be366b4-7891-46e6-89e5-c8c79ac042e9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:54.029243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be366b4-7891-46e6-89e5-c8c79ac042e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:54.029752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be366b4-7891-46e6-89e5-c8c79ac042e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:54.030676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be366b4-7891-46e6-89e5-c8c79ac042e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:54.032389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be366b4-7891-46e6-89e5-c8c79ac042e9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:54.050371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1098/2234] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:42:54 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:42:54.051730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:54.057813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:54.070744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2020ee66-432c-4a2d-937e-db75967569ba admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:54.076193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:54.077534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:54.078328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:54.078838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] PATH_INFO: `/v3/projects/3e77cfd4544c45468c57510de6034e38/users/82025bd919ce404b96491a19262d4281/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:54.100716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e77cfd4544c45468c57510de6034e38, user_id=82025bd919ce404b96491a19262d4281, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:54.104356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:54.124086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1137/2235] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:42:54 2022] PUT /identity/v3/projects/3e77cfd4544c45468c57510de6034e38/users/82025bd919ce404b96491a19262d4281/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:42:54.126343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8b109-7837-420a-8d93-af2cbe61df01 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:54.126926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8b109-7837-420a-8d93-af2cbe61df01 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:54.127366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8b109-7837-420a-8d93-af2cbe61df01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:54.147964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-5dc8b109-7837-420a-8d93-af2cbe61df01 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:42:54.151237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-5dc8b109-7837-420a-8d93-af2cbe61df01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:54.248624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:42:54.288992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1099/2236] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:42:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:42:54.299656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:42:54.301609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:42:54.388423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0747ed8b-ffc5-4d80-b42d-4a267c3e3eed admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:42:54.392785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:42:54.393646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:42:54.393934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:42:54.394213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:42:54.406692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:42:54.408350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:42:54.421028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1138/2237] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:42:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:43:01.097020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6c0d24-4502-4b10-8b6d-2653ae0e7a82 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:01.097514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6c0d24-4502-4b10-8b6d-2653ae0e7a82 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:01.097514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6c0d24-4502-4b10-8b6d-2653ae0e7a82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:01.144403 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bb6c0d24-4502-4b10-8b6d-2653ae0e7a82 None None] MFA Rules not processed for user `82025bd919ce404b96491a19262d4281`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:01.145640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bb6c0d24-4502-4b10-8b6d-2653ae0e7a82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:01.286308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1100/2238] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:43:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:01.298085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:01.316706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7d3d40-26a1-4b3d-a5a9-6554655a7f00 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:01.321882 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:01.322729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:01.323074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:01.323425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:01.324242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:01.326295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:01.376500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1139/2239] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:43:01 2022] POST /identity/v3/projects => generated 383 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:43:01.381014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6c0d24-4502-4b10-8b6d-2653ae0e7a82 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:01.396981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6c0d24-4502-4b10-8b6d-2653ae0e7a82 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:01.496278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:01.496710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:01.496893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:01.498445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:01.498445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:01.500688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:01.539485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1101/2240] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:43:01 2022] POST /identity/v3/users => generated 298 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:43:01.543357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:01.572509 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0af7ca5-4b9d-41ca-b264-19cb7aa65794 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:01.578340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:01.579536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:01.579924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:01.580363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:01.581827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:01.584625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:01.605336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1140/2241] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:43:01 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:43:01.606290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:01.618683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e307123-4cde-45ef-8e98-70773bc7e017 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:01.626685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:01.627893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:01.628184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:01.628473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] PATH_INFO: `/v3/projects/0b70d99ff0094ac6b018c6f8a907e31b/users/8da898ceacd44be996652faccba3381a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:01.661959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b70d99ff0094ac6b018c6f8a907e31b, user_id=8da898ceacd44be996652faccba3381a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:01.675639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:01.700312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1102/2242] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:43:01 2022] PUT /identity/v3/projects/0b70d99ff0094ac6b018c6f8a907e31b/users/8da898ceacd44be996652faccba3381a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:43:01.704778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:01.719556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc50bbf-3560-48d6-b813-0a7c37ba7f12 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:01.854024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9385a289-5ba5-4e82-b375-4465bc2180cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:01.854746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9385a289-5ba5-4e82-b375-4465bc2180cd admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:01.855048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9385a289-5ba5-4e82-b375-4465bc2180cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:01.855328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9385a289-5ba5-4e82-b375-4465bc2180cd admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:01.856168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9385a289-5ba5-4e82-b375-4465bc2180cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:01.857745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9385a289-5ba5-4e82-b375-4465bc2180cd admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:01.891352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1141/2243] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:43:01 2022] GET /identity/v3/roles => generated 1420 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:43:01.896393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:01.917557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0ced1e-feb0-4de7-9cd4-9a545936c490 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:01.928735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:01.931352 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:01.931962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:01.932424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] PATH_INFO: `/v3/projects/0b70d99ff0094ac6b018c6f8a907e31b/users/8da898ceacd44be996652faccba3381a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:01.936459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b70d99ff0094ac6b018c6f8a907e31b, user_id=8da898ceacd44be996652faccba3381a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:01.947592 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:01.971528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1103/2244] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:43:01 2022] PUT /identity/v3/projects/0b70d99ff0094ac6b018c6f8a907e31b/users/8da898ceacd44be996652faccba3381a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:43:09.438833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d756569-530c-419f-abdc-d1d11dc13ad8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:09.439432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d756569-530c-419f-abdc-d1d11dc13ad8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:09.439432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d756569-530c-419f-abdc-d1d11dc13ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:09.474370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2d756569-530c-419f-abdc-d1d11dc13ad8 None None] MFA Rules not processed for user `8da898ceacd44be996652faccba3381a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:09.475676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2d756569-530c-419f-abdc-d1d11dc13ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:09.519669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:43:09.573209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1142/2245] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:43:09 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:09.586861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:09.588606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:09.619304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:43:09.655788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d734665-6784-4f43-96b1-164b4a07b70a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:09.666806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fb6c2e2ddb3b4d488fa25fa8e258fec0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:09.667492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:09.667794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:09.668640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:09.799478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:09.800947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:09.815703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1104/2246] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:43:09 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:43:09.942682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ab1078-cf8e-4eb0-98c9-121d3eab5d84 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:09.943258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ab1078-cf8e-4eb0-98c9-121d3eab5d84 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:09.943746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ab1078-cf8e-4eb0-98c9-121d3eab5d84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:09.968958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b1ab1078-cf8e-4eb0-98c9-121d3eab5d84 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:09.971729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b1ab1078-cf8e-4eb0-98c9-121d3eab5d84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:10.095796 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1143/2247] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:43:09 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:10.108537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:10.110518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:10.207469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10357911-d820-4184-9e7f-f01998fc380d service glance] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:10.213179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79305d98-20d6-4d3b-8306-16665c5ebdf8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:10.213867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79305d98-20d6-4d3b-8306-16665c5ebdf8 service swift] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:10.214163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79305d98-20d6-4d3b-8306-16665c5ebdf8 service swift] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:10.214527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79305d98-20d6-4d3b-8306-16665c5ebdf8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:10.224453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79305d98-20d6-4d3b-8306-16665c5ebdf8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:10.226838 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79305d98-20d6-4d3b-8306-16665c5ebdf8 service swift] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:10.236467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1105/2248] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:43:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 11 14:43:15.089140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519417b6-79aa-4f16-94e7-208e1451d1ee None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.089745 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519417b6-79aa-4f16-94e7-208e1451d1ee None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.090115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519417b6-79aa-4f16-94e7-208e1451d1ee None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:15.092250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1144/2249] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:43:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:43:15.105322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c40274-ecef-4064-a462-41b565d76296 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.105840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c40274-ecef-4064-a462-41b565d76296 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.105840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c40274-ecef-4064-a462-41b565d76296 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:15.149667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-83c40274-ecef-4064-a462-41b565d76296 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:15.151554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-83c40274-ecef-4064-a462-41b565d76296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:15.372974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:43:15.384118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1106/2250] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:43:15 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:15.400404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519417b6-79aa-4f16-94e7-208e1451d1ee None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:15.417510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519417b6-79aa-4f16-94e7-208e1451d1ee None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:15.425350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739cf397-6f92-48bc-b64f-a7da00b804c2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fb6c2e2ddb3b4d488fa25fa8e258fec0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:15.426313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739cf397-6f92-48bc-b64f-a7da00b804c2 service glance] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.426842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739cf397-6f92-48bc-b64f-a7da00b804c2 service glance] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.427295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739cf397-6f92-48bc-b64f-a7da00b804c2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:15.455293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739cf397-6f92-48bc-b64f-a7da00b804c2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:15.461046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739cf397-6f92-48bc-b64f-a7da00b804c2 service glance] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:15.489061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1145/2251] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:43:15 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:43:15.652253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0176c6-318c-47c2-9983-67f2ceaffdd9 None None] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.652555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0176c6-318c-47c2-9983-67f2ceaffdd9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.652834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0176c6-318c-47c2-9983-67f2ceaffdd9 None None] PATH_INFO: `/` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:15.654481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1107/2252] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:43:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:43:15.663851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087685e-8c38-4cf7-8e55-b6076001a43a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.664145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087685e-8c38-4cf7-8e55-b6076001a43a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.664475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087685e-8c38-4cf7-8e55-b6076001a43a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:15.697413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c087685e-8c38-4cf7-8e55-b6076001a43a None None] MFA Rules not processed for user `ac5254fb5eff4108be9ac3992e05a99c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:15.698883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c087685e-8c38-4cf7-8e55-b6076001a43a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:15.786137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1146/2253] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:43:15 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:15.801189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0176c6-318c-47c2-9983-67f2ceaffdd9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:15.823595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0176c6-318c-47c2-9983-67f2ceaffdd9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:15.827543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f1415d-97bb-4837-9fc9-daa0ef645892 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:15.828217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f1415d-97bb-4837-9fc9-daa0ef645892 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.828507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f1415d-97bb-4837-9fc9-daa0ef645892 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.828894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f1415d-97bb-4837-9fc9-daa0ef645892 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:15.846347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f1415d-97bb-4837-9fc9-daa0ef645892 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:15.847858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f1415d-97bb-4837-9fc9-daa0ef645892 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:15.868865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1108/2254] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:43:15 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:43:15.973538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eef15d-e8e2-4401-984a-fac9c31705fb None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.973860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eef15d-e8e2-4401-984a-fac9c31705fb None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.974170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eef15d-e8e2-4401-984a-fac9c31705fb None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:15.975800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1147/2255] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:43:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:43:15.982419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:15.982680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:15.982963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:16.009325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] MFA Rules not processed for user `2165eacee8954dce975ba81ba8ac4232`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:16.010597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:16.021487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1109/2256] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:43:15 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:43:16.034154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eef15d-e8e2-4401-984a-fac9c31705fb None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:16.036404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-27eef15d-e8e2-4401-984a-fac9c31705fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:16.066864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eef15d-e8e2-4401-984a-fac9c31705fb None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:16.075393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:16.075636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:16.075837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:16.076121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:16.099460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:16.100987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:16.125444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1148/2257] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:43:16 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:43:18.537419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:18.538965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:18.751674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9b40b1-58c2-4630-8bb6-cd99c38704c4 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:18.761827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001a3e36-79ce-4dde-b79c-97c1b360598d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:18.762920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001a3e36-79ce-4dde-b79c-97c1b360598d service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:18.763442 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001a3e36-79ce-4dde-b79c-97c1b360598d service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:18.763938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001a3e36-79ce-4dde-b79c-97c1b360598d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:18.769432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-001a3e36-79ce-4dde-b79c-97c1b360598d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:18.794883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001a3e36-79ce-4dde-b79c-97c1b360598d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:18.796797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001a3e36-79ce-4dde-b79c-97c1b360598d service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:18.807895 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:18.809367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:18.813404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1110/2258] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:43:18 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:43:18.852591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf90049-7185-4314-a350-9f4ef5707a9e service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:18.880002 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:18.881111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:18.882871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:18.883329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:18.893807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:18.895756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:18.907636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1149/2259] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:43:18 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:43:21.804232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc6f6b4-e49f-4c06-84f1-1b315b3dd277 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:21.804850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc6f6b4-e49f-4c06-84f1-1b315b3dd277 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:21.805049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc6f6b4-e49f-4c06-84f1-1b315b3dd277 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:21.848362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1dc6f6b4-e49f-4c06-84f1-1b315b3dd277 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:21.849596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1dc6f6b4-e49f-4c06-84f1-1b315b3dd277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:21.872252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1111/2260] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:43:21 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:43:21.888299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:21.907545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090e4f57-7697-4e46-950f-4c5e7b9aef93 service placement] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:21.923245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf812ff-a4ee-4d75-b961-0938700c4c50 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:21.923920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf812ff-a4ee-4d75-b961-0938700c4c50 service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:21.924393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf812ff-a4ee-4d75-b961-0938700c4c50 service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:21.924926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf812ff-a4ee-4d75-b961-0938700c4c50 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:21.965053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf812ff-a4ee-4d75-b961-0938700c4c50 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:21.967276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf812ff-a4ee-4d75-b961-0938700c4c50 service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:21.980293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1150/2261] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:43:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:43:22.728993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33915a0-c899-4dd2-b55e-7d8159ca4ab0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:22.729624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33915a0-c899-4dd2-b55e-7d8159ca4ab0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:22.729974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33915a0-c899-4dd2-b55e-7d8159ca4ab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:22.762573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c33915a0-c899-4dd2-b55e-7d8159ca4ab0 None None] MFA Rules not processed for user `82025bd919ce404b96491a19262d4281`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:22.764133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c33915a0-c899-4dd2-b55e-7d8159ca4ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:22.773302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1112/2262] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:43:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:31.735109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0496e6f-578b-43b1-b9cb-9d1aeaaf1350 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:31.735702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0496e6f-578b-43b1-b9cb-9d1aeaaf1350 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:31.735702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0496e6f-578b-43b1-b9cb-9d1aeaaf1350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:31.755330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e0496e6f-578b-43b1-b9cb-9d1aeaaf1350 None None] MFA Rules not processed for user `b7457a38a0594e8da4c053a5c6a4aa15`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:31.756596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e0496e6f-578b-43b1-b9cb-9d1aeaaf1350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:31.786712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:43:31.818368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1151/2263] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:43:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:32.750017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=48 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:43:34.671741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090601af-00a5-43f6-bbdd-01ef6c3700ed None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:34.673346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090601af-00a5-43f6-bbdd-01ef6c3700ed None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:34.673826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090601af-00a5-43f6-bbdd-01ef6c3700ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:34.695519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-090601af-00a5-43f6-bbdd-01ef6c3700ed None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:34.698234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-090601af-00a5-43f6-bbdd-01ef6c3700ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:34.711064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1113/2264] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:43:34 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:34.727649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0496e6f-578b-43b1-b9cb-9d1aeaaf1350 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:34.761132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0496e6f-578b-43b1-b9cb-9d1aeaaf1350 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:34.770184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:34.771137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:34.771584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:34.771959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:34.782070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:34.783796 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:34.793849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1152/2265] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:43:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:43:44.689513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:43:44.790245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=17 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:43:53.589726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f6493-af98-4e76-9005-c822321b8852 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:53.590227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f6493-af98-4e76-9005-c822321b8852 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:53.590335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f6493-af98-4e76-9005-c822321b8852 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:53.611451 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ae3f6493-af98-4e76-9005-c822321b8852 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:43:53.612668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ae3f6493-af98-4e76-9005-c822321b8852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:43:53.620005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1114/2266] 10.222.0.72 () {58 vars in 967 bytes} [Mon Jul 11 14:43:53 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:43:53.643025 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:43:53.654244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db211d1b-eb82-4314-9798-654839bf42e6 service swift] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:43:53.659770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd353c7-9ec1-4719-9949-28b1b202c880 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:43:53.660490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd353c7-9ec1-4719-9949-28b1b202c880 service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:43:53.660786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd353c7-9ec1-4719-9949-28b1b202c880 service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:43:53.661069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd353c7-9ec1-4719-9949-28b1b202c880 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:43:53.676077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd353c7-9ec1-4719-9949-28b1b202c880 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:43:53.677762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd353c7-9ec1-4719-9949-28b1b202c880 service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:43:53.686891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1153/2267] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:43:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:44:03.604563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:44:03.683417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=6 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:44:17.602915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f6493-af98-4e76-9005-c822321b8852 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:44:17.612243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f6493-af98-4e76-9005-c822321b8852 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:44:17.616329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889f5f33-c097-469c-b472-df0d28fa558b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:44:17.617032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889f5f33-c097-469c-b472-df0d28fa558b service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:44:17.617286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889f5f33-c097-469c-b472-df0d28fa558b service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:44:17.617564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889f5f33-c097-469c-b472-df0d28fa558b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:44:17.627564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889f5f33-c097-469c-b472-df0d28fa558b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:44:17.629001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889f5f33-c097-469c-b472-df0d28fa558b service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:44:17.641153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1115/2268] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:44:17 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:44:25.249092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:44:25.249615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:44:25.249713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:44:25.267650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] MFA Rules not processed for user `3ae56fd1172f4bf8806eefc6e8877fef`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:44:25.268862 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:44:25.326456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1154/2269] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:44:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:44:27.635861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=6 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:44:35.318769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=25 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:44:48.208491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce1039f-dd8a-47fe-90b8-b3480ca3ce76 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:44:48.209086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce1039f-dd8a-47fe-90b8-b3480ca3ce76 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:44:48.209086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce1039f-dd8a-47fe-90b8-b3480ca3ce76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:44:48.228101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7ce1039f-dd8a-47fe-90b8-b3480ca3ce76 None None] MFA Rules not processed for user `fa478fb4b41e49c5b1f9291f925cca23`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:44:48.229333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7ce1039f-dd8a-47fe-90b8-b3480ca3ce76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:44:48.302410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1116/2270] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:44:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:44:54.818836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:44:54.820311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:44:54.887741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee353bd9-79e7-44c0-9af6-d100e208c3da None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:44:54.892345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:44:54.893227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:44:54.893518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:44:54.893802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] PATH_INFO: `/v3/users/b7457a38a0594e8da4c053a5c6a4aa15` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:44:54.895038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7457a38a0594e8da4c053a5c6a4aa15)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:44:54.897563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:44:54.945881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7457a38a0594e8da4c053a5c6a4aa15'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:54.953801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7457a38a0594e8da4c053a5c6a4aa15'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:54.954523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7457a38a0594e8da4c053a5c6a4aa15'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:54.962684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7457a38a0594e8da4c053a5c6a4aa15'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:54.977296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-5e9cce4a-b790-4ad3-b1ad-b311a6e44b5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7457a38a0594e8da4c053a5c6a4aa15'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:54.984237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1155/2271] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:44:54 2022] DELETE /identity/v3/users/b7457a38a0594e8da4c053a5c6a4aa15 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:44:55.182482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce1039f-dd8a-47fe-90b8-b3480ca3ce76 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:44:55.201069 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce1039f-dd8a-47fe-90b8-b3480ca3ce76 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:44:55.286209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:44:55.286209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:44:55.286209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:44:55.288011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] PATH_INFO: `/v3/projects/8e0ccc1e4683478889d90814a1305507` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:44:55.288163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e0ccc1e4683478889d90814a1305507)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:44:55.290502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:44:55.371075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e0ccc1e4683478889d90814a1305507', 'request_id': 'req-8b082b53-8a0b-4801-b6c9-220bc8354904'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:55.377597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e0ccc1e4683478889d90814a1305507', 'request_id': 'req-8b082b53-8a0b-4801-b6c9-220bc8354904'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:55.397823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e0ccc1e4683478889d90814a1305507', 'request_id': 'req-8b082b53-8a0b-4801-b6c9-220bc8354904'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:55.413376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] The token cache is being invalidate because project 8e0ccc1e4683478889d90814a1305507 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:44:55.413678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:55.414377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:44:55.422477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1117/2272] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:44:55 2022] DELETE /identity/v3/projects/8e0ccc1e4683478889d90814a1305507 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:45:03.737858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:03.738374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:03.738506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:03.757749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] MFA Rules not processed for user `99a62e49e6d249c2a169ef7122e762ec`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:45:03.759006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:03.814247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1156/2273] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:45:03 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:45:05.409760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=76,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:45:12.817411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:12.818890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:12.877065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b082b53-8a0b-4801-b6c9-220bc8354904 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:12.881300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab183c5-0347-4731-94f9-25b3f43897a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:12.881964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab183c5-0347-4731-94f9-25b3f43897a3 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:12.882248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab183c5-0347-4731-94f9-25b3f43897a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:12.882557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab183c5-0347-4731-94f9-25b3f43897a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:12.884391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3ab183c5-0347-4731-94f9-25b3f43897a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:12.958776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab183c5-0347-4731-94f9-25b3f43897a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:12.960642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab183c5-0347-4731-94f9-25b3f43897a3 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:12.972165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1118/2274] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:12 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:13.234315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:13.235805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:13.247710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b7af47-6272-4867-8cba-9924c14fc1ed None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:13.251367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:13.252008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:13.252296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:13.252576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:13.254361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:13.268456 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:13.269888 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:13.281660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1157/2275] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:13 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:22.966993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=54 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:45:23.276931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=84,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:45:24.222405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:24.223011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:24.223133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:24.241670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] MFA Rules not processed for user `0f8e300cd96e446db7430044e69d997a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:45:24.243213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:24.317589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1119/2276] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:45:24 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:45:24.898171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:24.911626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210143ac-65f4-4d68-a3e0-dabda73937a8 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:24.915642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7118e36-bac5-4544-8d30-77a9bc2abe0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:24.916386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7118e36-bac5-4544-8d30-77a9bc2abe0a service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:24.916673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7118e36-bac5-4544-8d30-77a9bc2abe0a service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:24.916951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7118e36-bac5-4544-8d30-77a9bc2abe0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:24.919089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d7118e36-bac5-4544-8d30-77a9bc2abe0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:24.931030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7118e36-bac5-4544-8d30-77a9bc2abe0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:24.932637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7118e36-bac5-4544-8d30-77a9bc2abe0a service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:24.943275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1158/2277] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:24 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:25.544644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:25.546082 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:25.649537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c495d0-8a86-4af6-bb86-13afa891e2bd None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:25.654567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:25.655233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:25.655523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:25.655808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:25.657673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:25.672057 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:25.673820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:25.686869 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1120/2278] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:25 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:26.463340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbad6c-9521-435c-b019-60874aef9666 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:26.463859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbad6c-9521-435c-b019-60874aef9666 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:26.463966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbad6c-9521-435c-b019-60874aef9666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:26.481564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-38bbad6c-9521-435c-b019-60874aef9666 None None] MFA Rules not processed for user `467992b53c4749858fd8b0ccd2db2393`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:45:26.482828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-38bbad6c-9521-435c-b019-60874aef9666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:26.541286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1159/2279] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:45:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:45:27.264173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:27.266000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:27.285321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8de201-c4de-440f-9188-d48517d89a17 service placement] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:27.292865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:27.293540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:27.293821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:27.294318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:27.296478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:27.395578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:27.401649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:27.419224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1121/2280] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:32.165913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbad6c-9521-435c-b019-60874aef9666 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:32.179248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbad6c-9521-435c-b019-60874aef9666 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:32.184787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:32.185455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:32.185749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:32.186036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:32.187925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:32.199405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:32.201046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:32.212468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1160/2281] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:32 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:34.513532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:34.515132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:34.556975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:45:34.581087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfb47a-3028-48fb-a63d-a1836ca7e194 service nova] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:34.585900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:34.586613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:34.586904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:34.587186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:34.589042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:34.689143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:34.691191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:34.702846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1122/2282] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:34 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:36.932163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:36.953238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ede1cc-4426-4102-a2d9-d118d5432e7c service nova] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:36.960285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:36.961631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:36.962099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:36.962650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:36.966156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:36.983657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:36.986710 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:37.007907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1161/2283] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:36 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:41.144210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:41.146953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:41.165486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019a9e5-77d3-4447-a16c-d6767116d5e7 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:41.173364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c5a75f-0949-4714-b044-3f12444004f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:41.174185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c5a75f-0949-4714-b044-3f12444004f5 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:41.174693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c5a75f-0949-4714-b044-3f12444004f5 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:41.175142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c5a75f-0949-4714-b044-3f12444004f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:41.177633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-16c5a75f-0949-4714-b044-3f12444004f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:41.191834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c5a75f-0949-4714-b044-3f12444004f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:41.193623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c5a75f-0949-4714-b044-3f12444004f5 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:41.206015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1123/2284] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:41 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:47.000502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=43 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:45:51.167827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:51.192533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df275318-62e8-4674-8f56-76a3cdcd42d4 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:51.200989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=39 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:45:51.203246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:51.205067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:51.205939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:51.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:51.210709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:51.235380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:51.238554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:51.261299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1162/2285] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:51 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:45:52.987031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a6f21-957b-401a-83a3-9645e648cf7f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:52.987904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a6f21-957b-401a-83a3-9645e648cf7f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:52.988275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a6f21-957b-401a-83a3-9645e648cf7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:53.011460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-490a6f21-957b-401a-83a3-9645e648cf7f None None] MFA Rules not processed for user `3ae56fd1172f4bf8806eefc6e8877fef`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:45:53.013015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-490a6f21-957b-401a-83a3-9645e648cf7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:53.024099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1124/2286] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:45:52 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:45:53.414175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:45:53.424692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1f041d-3c27-481b-8777-d76a19a6067e service nova] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:45:53.428769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:45:53.429447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:45:53.429728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:45:53.430008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:45:53.432001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:45:53.444410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:45:53.445958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:45:53.455392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1163/2287] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:45:53 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:02.105510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a6f21-957b-401a-83a3-9645e648cf7f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:02.121470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a6f21-957b-401a-83a3-9645e648cf7f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:02.128768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:02.129701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:02.130078 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:02.130496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:02.132965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:02.147416 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:02.149209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:02.162392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1125/2288] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:02 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:02.986476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:02.997437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de36f98b-e7a7-4d71-bea6-60e25037c22b service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:03.002348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52208681-fb14-4491-82e7-4030688a5000 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:03.003214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52208681-fb14-4491-82e7-4030688a5000 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:03.003646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52208681-fb14-4491-82e7-4030688a5000 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:03.003948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52208681-fb14-4491-82e7-4030688a5000 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:03.006185 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-52208681-fb14-4491-82e7-4030688a5000 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:03.099289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52208681-fb14-4491-82e7-4030688a5000 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:03.100784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52208681-fb14-4491-82e7-4030688a5000 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:03.112590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1164/2289] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:06.275456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:06.295544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a21115-f7d3-4b63-9e39-3bf6a1150047 service nova] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:06.313476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee79c96-82cb-43b8-b655-40d875debac5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:06.316050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee79c96-82cb-43b8-b655-40d875debac5 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:06.317461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee79c96-82cb-43b8-b655-40d875debac5 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:06.319727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee79c96-82cb-43b8-b655-40d875debac5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:06.329318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bee79c96-82cb-43b8-b655-40d875debac5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:06.359455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee79c96-82cb-43b8-b655-40d875debac5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:06.362496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee79c96-82cb-43b8-b655-40d875debac5 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:06.389386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1126/2290] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:08.061881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bc3ce-2459-487a-888c-05bf09418750 None None] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:08.062552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bc3ce-2459-487a-888c-05bf09418750 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:08.062552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bc3ce-2459-487a-888c-05bf09418750 None None] PATH_INFO: `/` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:08.064165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1165/2291] 10.222.0.72 () {54 vars in 921 bytes} [Mon Jul 11 14:46:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 11 14:46:08.081359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:08.082921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:08.082921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:08.104673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] MFA Rules not processed for user `5aabd82e0fd04345b63c346660e0da54`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:08.106013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:08.115268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1127/2292] 10.222.0.72 () {58 vars in 1018 bytes} [Mon Jul 11 14:46:08 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:08.177740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bc3ce-2459-487a-888c-05bf09418750 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:08.200187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bc3ce-2459-487a-888c-05bf09418750 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:08.208068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:08.209246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:08.209725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:08.210410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:08.210945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:08.211452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:08.226718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b3cc63-0c0b-46b4-96f8-cc9ff1a61a52 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:08.230643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:08.233862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:08.234081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:08.235246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:08.235708 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:08.236169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] PATH_INFO: `/v3/users/fa478fb4b41e49c5b1f9291f925cca23` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:08.237679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa478fb4b41e49c5b1f9291f925cca23)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:08.240160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:08.254186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1166/2293] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:08 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:08.318532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa478fb4b41e49c5b1f9291f925cca23'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.327547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa478fb4b41e49c5b1f9291f925cca23'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.337689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa478fb4b41e49c5b1f9291f925cca23'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.338493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa478fb4b41e49c5b1f9291f925cca23'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.355429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9b36299a-43c0-4f5b-aedb-32db94995aa0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa478fb4b41e49c5b1f9291f925cca23'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.363129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1128/2294] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:08 2022] DELETE /identity/v3/users/fa478fb4b41e49c5b1f9291f925cca23 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:08.597664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:08.617931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5825a7-da33-4a50-b5dc-0eed548471f0 service placement] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:08.696643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:08.697565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:08.697850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:08.698128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] PATH_INFO: `/v3/projects/98966d44ea88477b94215761cdc92ea7` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:08.699536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] RBAC: Authorizing `identity:delete_project(project_id=98966d44ea88477b94215761cdc92ea7)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:08.701499 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:08.762245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98966d44ea88477b94215761cdc92ea7', 'request_id': 'req-4b1687f0-bfe9-4347-b7d4-38c4d4059818'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.772077 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98966d44ea88477b94215761cdc92ea7', 'request_id': 'req-4b1687f0-bfe9-4347-b7d4-38c4d4059818'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.774003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:08.783269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98966d44ea88477b94215761cdc92ea7', 'request_id': 'req-4b1687f0-bfe9-4347-b7d4-38c4d4059818'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.806788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] The token cache is being invalidate because project 98966d44ea88477b94215761cdc92ea7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:08.807328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.808690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-4b1687f0-bfe9-4347-b7d4-38c4d4059818 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:08.811758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1167/2295] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:08 2022] DELETE /identity/v3/projects/98966d44ea88477b94215761cdc92ea7 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:08.818370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8226f4-7fac-4ee4-881d-9c3657a84ec2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:08.818698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8226f4-7fac-4ee4-881d-9c3657a84ec2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:08.818986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8226f4-7fac-4ee4-881d-9c3657a84ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:08.847322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fe8226f4-7fac-4ee4-881d-9c3657a84ec2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:08.849741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fe8226f4-7fac-4ee4-881d-9c3657a84ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:08.945691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1129/2296] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:08.955497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e932614-b88b-4767-910d-9ddedf9dd1fe None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:08.956044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e932614-b88b-4767-910d-9ddedf9dd1fe None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:08.956556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e932614-b88b-4767-910d-9ddedf9dd1fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:08.989534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9e932614-b88b-4767-910d-9ddedf9dd1fe None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:08.991708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9e932614-b88b-4767-910d-9ddedf9dd1fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:09.010423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1168/2297] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:09.015302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8226f4-7fac-4ee4-881d-9c3657a84ec2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.026422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8226f4-7fac-4ee4-881d-9c3657a84ec2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.031733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.032423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.032711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.032990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.033746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:46:09.034062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.035835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.049647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1130/2298] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:46:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:46:09.055391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e932614-b88b-4767-910d-9ddedf9dd1fe None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.070971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e932614-b88b-4767-910d-9ddedf9dd1fe None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.077974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.079030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.079500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.080006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.081239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.084966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.126284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1169/2299] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:09 2022] POST /identity/v3/projects => generated 365 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:09.132512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.144252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d455e-1b32-4dba-b08b-d172e5ed8e3f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.175761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:09.244541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.245502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.245966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.246292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.247114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.249238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.280964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1131/2300] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:09 2022] POST /identity/v3/users => generated 283 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:09.285729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.308995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1285729e-c835-403d-b92e-5678f72d13aa admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.323087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.324004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.324493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.324897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.325788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.327823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.344080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1170/2301] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:09 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:09.350510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.361732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879718e-d396-4920-9266-6b5aca4a690f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.367206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93ac5d4-7f90-40c5-b5fc-20f40c4bb0eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.367906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93ac5d4-7f90-40c5-b5fc-20f40c4bb0eb admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.368189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93ac5d4-7f90-40c5-b5fc-20f40c4bb0eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.368473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93ac5d4-7f90-40c5-b5fc-20f40c4bb0eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.369229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93ac5d4-7f90-40c5-b5fc-20f40c4bb0eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.370987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93ac5d4-7f90-40c5-b5fc-20f40c4bb0eb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.384850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1132/2302] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:09.389678 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.402016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b422152-34dd-4d3b-92a3-156bf394d9f0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.407006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.407930 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.408219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.408495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] PATH_INFO: `/v3/projects/22c77fbadf4949098c1d01788485ac77/users/6cfb677adc5243e9aa73e4a0c306bce4/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.431854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=22c77fbadf4949098c1d01788485ac77, user_id=6cfb677adc5243e9aa73e4a0c306bce4, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.441385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.475295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1171/2303] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:09 2022] PUT /identity/v3/projects/22c77fbadf4949098c1d01788485ac77/users/6cfb677adc5243e9aa73e4a0c306bce4/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:09.492507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c9a85-f320-461b-931a-2cee38d15d66 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.493031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c9a85-f320-461b-931a-2cee38d15d66 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.493549 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c9a85-f320-461b-931a-2cee38d15d66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.545771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f47c9a85-f320-461b-931a-2cee38d15d66 None None] MFA Rules not processed for user `6cfb677adc5243e9aa73e4a0c306bce4`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:09.547892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f47c9a85-f320-461b-931a-2cee38d15d66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:09.653987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1133/2304] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:09 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:09.659376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.737393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315893e-0267-4daa-b289-eb224ee563ff admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.743191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.743531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.743699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.743940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.744671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.746742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.805976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1172/2305] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:09 2022] POST /identity/v3/projects => generated 365 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:09.811427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c9a85-f320-461b-931a-2cee38d15d66 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.823538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c9a85-f320-461b-931a-2cee38d15d66 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.895079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.895837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.896096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.896864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.897613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.899664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.928406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1134/2306] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:09 2022] POST /identity/v3/users => generated 289 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:09.933451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.949265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1060803b-380f-4718-b8f7-48cd0bebf55e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:09.955155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:09.955850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:09.956137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:09.956418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:09.957439 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:09.959134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:09.979172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1173/2307] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:09 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:09.980663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:09.993917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708e49c1-7db1-46b7-9692-82c0e5323a01 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:10.000469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:10.001724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:10.002199 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:10.002762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] PATH_INFO: `/v3/projects/f1cad068123b404c9af0190ace003671/users/71715b5703704767af17ffddf52943e6/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:10.021378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1cad068123b404c9af0190ace003671, user_id=71715b5703704767af17ffddf52943e6, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:10.024270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:10.027036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:10.042215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1135/2308] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:09 2022] PUT /identity/v3/projects/f1cad068123b404c9af0190ace003671/users/71715b5703704767af17ffddf52943e6/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:10.047087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:10.065129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91755c1e-0934-4706-b794-690cc32383d4 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:10.133781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:10.145906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb79a20-6e2b-4517-a91a-792589123e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:10.146788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb79a20-6e2b-4517-a91a-792589123e8e admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:10.147291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb79a20-6e2b-4517-a91a-792589123e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:10.148067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb79a20-6e2b-4517-a91a-792589123e8e admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:10.149349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb79a20-6e2b-4517-a91a-792589123e8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:10.151818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb79a20-6e2b-4517-a91a-792589123e8e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:10.165780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1174/2309] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:10 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:10.170156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:10.185472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747adeac-24f1-4e1c-9863-2e1bf4eddb21 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:10.194116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:10.195623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:10.196123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:10.196626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] PATH_INFO: `/v3/projects/f1cad068123b404c9af0190ace003671/users/71715b5703704767af17ffddf52943e6/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:10.200316 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1cad068123b404c9af0190ace003671, user_id=71715b5703704767af17ffddf52943e6, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:10.205559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:10.225559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1136/2310] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:10 2022] PUT /identity/v3/projects/f1cad068123b404c9af0190ace003671/users/71715b5703704767af17ffddf52943e6/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:10.242039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78a2ee-4d75-4eaa-ae1c-0f47bcf3e096 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:10.242469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78a2ee-4d75-4eaa-ae1c-0f47bcf3e096 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:10.242972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78a2ee-4d75-4eaa-ae1c-0f47bcf3e096 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:10.291832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2c78a2ee-4d75-4eaa-ae1c-0f47bcf3e096 None None] MFA Rules not processed for user `71715b5703704767af17ffddf52943e6`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:10.293912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2c78a2ee-4d75-4eaa-ae1c-0f47bcf3e096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:10.412893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1175/2311] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:10 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:10.428108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:10.431049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:10.532312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed64df2-f5b9-4576-999d-a36994f618ee admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:10.537334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff1dbb3-55e8-46cf-baec-63ba2deaf34d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:10.538089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1dbb3-55e8-46cf-baec-63ba2deaf34d service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:10.538388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1dbb3-55e8-46cf-baec-63ba2deaf34d service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:10.538746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1dbb3-55e8-46cf-baec-63ba2deaf34d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:10.551865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff1dbb3-55e8-46cf-baec-63ba2deaf34d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:10.553799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff1dbb3-55e8-46cf-baec-63ba2deaf34d service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:10.568084 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1137/2312] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:10 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:11.233113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78a2ee-4d75-4eaa-ae1c-0f47bcf3e096 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:11.362755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78a2ee-4d75-4eaa-ae1c-0f47bcf3e096 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:11.372621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:11.374344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:11.374918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:11.375450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] PATH_INFO: `/v3/users/6cfb677adc5243e9aa73e4a0c306bce4` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:11.377821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cfb677adc5243e9aa73e4a0c306bce4)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:11.382459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:11.493212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cfb677adc5243e9aa73e4a0c306bce4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:11.509742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cfb677adc5243e9aa73e4a0c306bce4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:11.511070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cfb677adc5243e9aa73e4a0c306bce4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:11.528907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cfb677adc5243e9aa73e4a0c306bce4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:11.555723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:11.558616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cfb677adc5243e9aa73e4a0c306bce4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:11.575764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1176/2313] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:11 2022] DELETE /identity/v3/users/6cfb677adc5243e9aa73e4a0c306bce4 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:11.587780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:11.593297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:11.594349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:11.625533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:11.627227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:11.752588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1138/2314] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:11.760304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:11.762818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:11.859351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86103ea-405d-4950-bc5f-8ee9fa2ded75 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:11.864200 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:11.864880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:11.865240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:11.865535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:11.881591 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:11.883172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:11.896665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1177/2315] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:12.311174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:12.313847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:12.353046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65dd37-5330-423f-93cf-d1f736dc8ce2 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:12.368098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:12.369188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:12.369604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:12.369996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] PATH_INFO: `/v3/projects/22c77fbadf4949098c1d01788485ac77` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:12.372011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] RBAC: Authorizing `identity:delete_project(project_id=22c77fbadf4949098c1d01788485ac77)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:12.374939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:12.461922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '22c77fbadf4949098c1d01788485ac77', 'request_id': 'req-30c0613d-e859-4a37-83fa-e3eb43e4edba'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.468960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '22c77fbadf4949098c1d01788485ac77', 'request_id': 'req-30c0613d-e859-4a37-83fa-e3eb43e4edba'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.477603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '22c77fbadf4949098c1d01788485ac77', 'request_id': 'req-30c0613d-e859-4a37-83fa-e3eb43e4edba'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.491261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] The token cache is being invalidate because project 22c77fbadf4949098c1d01788485ac77 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:12.491567 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.492404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.494520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1139/2316] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:12 2022] DELETE /identity/v3/projects/22c77fbadf4949098c1d01788485ac77 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:12.499206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:12.517832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270383d5-a977-45ed-93b6-9dda9e71cc62 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:12.611441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:12.612515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:12.612894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:12.613755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] PATH_INFO: `/v3/users/71715b5703704767af17ffddf52943e6` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:12.615317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] RBAC: Authorizing `identity:delete_user(user_id=71715b5703704767af17ffddf52943e6)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:12.617621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:12.685395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71715b5703704767af17ffddf52943e6'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.693951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71715b5703704767af17ffddf52943e6'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.694733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71715b5703704767af17ffddf52943e6'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.702988 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71715b5703704767af17ffddf52943e6'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.718493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-94da065a-0e70-47e2-926c-fd0f21cfe67a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71715b5703704767af17ffddf52943e6'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:12.726042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1178/2317] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:12 2022] DELETE /identity/v3/users/71715b5703704767af17ffddf52943e6 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:13.091619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:13.093736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:13.111885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0613d-e859-4a37-83fa-e3eb43e4edba admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:13.117600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:INSERT=3,keystone:SELECT=90,keystone:DELETE=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:13.195366 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:13.196424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.196833 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.197249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] PATH_INFO: `/v3/projects/f1cad068123b404c9af0190ace003671` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.198698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1cad068123b404c9af0190ace003671)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:13.201109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:13.263962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1cad068123b404c9af0190ace003671', 'request_id': 'req-6328fd30-6366-4d65-b41e-84f05e5981ba'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:13.274638 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1cad068123b404c9af0190ace003671', 'request_id': 'req-6328fd30-6366-4d65-b41e-84f05e5981ba'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:13.289281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1cad068123b404c9af0190ace003671', 'request_id': 'req-6328fd30-6366-4d65-b41e-84f05e5981ba'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:13.304157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] The token cache is being invalidate because project f1cad068123b404c9af0190ace003671 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:13.304566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:13.305303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6328fd30-6366-4d65-b41e-84f05e5981ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:13.307018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1140/2318] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:13 2022] DELETE /identity/v3/projects/f1cad068123b404c9af0190ace003671 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:13.312794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fab46-b484-455e-9377-87b63a11f46a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.313088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fab46-b484-455e-9377-87b63a11f46a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.313326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fab46-b484-455e-9377-87b63a11f46a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.334028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-6a1fab46-b484-455e-9377-87b63a11f46a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:13.335243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-6a1fab46-b484-455e-9377-87b63a11f46a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:13.350585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:13.429633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1179/2319] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:13.438913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac491a6e-9ae9-44dc-92f8-751a16094c8a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.439168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac491a6e-9ae9-44dc-92f8-751a16094c8a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.439453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac491a6e-9ae9-44dc-92f8-751a16094c8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.460801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ac491a6e-9ae9-44dc-92f8-751a16094c8a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:13.462125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ac491a6e-9ae9-44dc-92f8-751a16094c8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:13.476284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1141/2320] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:13.481396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fab46-b484-455e-9377-87b63a11f46a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:13.494516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fab46-b484-455e-9377-87b63a11f46a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:13.500310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:13.501390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.501790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.502255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.503234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:46:13.503659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:13.506162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:13.528417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1180/2321] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:46:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:46:13.533669 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac491a6e-9ae9-44dc-92f8-751a16094c8a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:13.550452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac491a6e-9ae9-44dc-92f8-751a16094c8a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:13.559319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:13.560489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.561002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.561505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.562878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:13.567388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:13.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1142/2322] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:13 2022] POST /identity/v3/projects => generated 363 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:13.616958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:13.631105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d4b48d-e12c-4a9a-9dd3-9ed0c4f4aaa6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:13.739461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:13.740206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.740530 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.740805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.741575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:13.743693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:13.768156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1181/2323] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:13 2022] POST /identity/v3/users => generated 282 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:13.773229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:13.786542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619d84ea-5352-491a-9b47-59f8e24de390 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:13.793030 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:13.793742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.794921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.795401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.796953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:13.799208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:13.816974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1143/2324] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:13 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:13.821417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:13.834250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a561c-d39d-4158-882d-9ea1009847bb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:13.839928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f7288b-6404-41a9-a97b-0c4213231859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:13.840657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f7288b-6404-41a9-a97b-0c4213231859 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.840921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f7288b-6404-41a9-a97b-0c4213231859 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.841197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f7288b-6404-41a9-a97b-0c4213231859 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.841964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f7288b-6404-41a9-a97b-0c4213231859 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:13.843709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f7288b-6404-41a9-a97b-0c4213231859 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:13.856944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1182/2325] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:13 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:13.863210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:13.878265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863be898-3991-4a5b-aa9e-ce9db3c74fdb admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:13.886235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:13.887806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.888310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.888829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] PATH_INFO: `/v3/projects/2ff621396c154c4f8767b25311b1c848/users/8bbb3a6e971043fdbf77a997743474f3/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.917090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ff621396c154c4f8767b25311b1c848, user_id=8bbb3a6e971043fdbf77a997743474f3, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:13.922590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:13.941808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1144/2326] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:13 2022] PUT /identity/v3/projects/2ff621396c154c4f8767b25311b1c848/users/8bbb3a6e971043fdbf77a997743474f3/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:13.952776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18100d1b-b443-439a-93dc-bcd4c15408d2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:13.953097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18100d1b-b443-439a-93dc-bcd4c15408d2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:13.953373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18100d1b-b443-439a-93dc-bcd4c15408d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:13.987963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-18100d1b-b443-439a-93dc-bcd4c15408d2 None None] MFA Rules not processed for user `8bbb3a6e971043fdbf77a997743474f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:13.990023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-18100d1b-b443-439a-93dc-bcd4c15408d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:14.068349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1183/2327] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:13 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:14.073051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:14.110386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:14.156008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d5703-d0a8-4a97-8e9e-3cff8222374f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:14.161378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:14.162256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.162649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.163563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.164301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:14.166263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:14.198462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1145/2328] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:14 2022] POST /identity/v3/projects => generated 361 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:14.203205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18100d1b-b443-439a-93dc-bcd4c15408d2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:14.215840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18100d1b-b443-439a-93dc-bcd4c15408d2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:14.226477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:14.292196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:14.293013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.293903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.293903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.294486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:14.296696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:14.335945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1184/2329] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:14 2022] POST /identity/v3/users => generated 287 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:14.341340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:14.356357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae1e397-40ec-4de8-800e-13b1d6fca87f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:14.363731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:14.364847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.365313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.365806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.367263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:14.370125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:14.383572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1146/2330] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:14 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:14.390800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:14.408407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b86a843-2f35-4313-841c-57d6da815c67 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:14.416292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:14.417908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.418267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.418749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] PATH_INFO: `/v3/projects/b8f93a7f750346e195ae6543fda1d705/users/4e7be52714b342beaefe916553ca7714/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.438574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8f93a7f750346e195ae6543fda1d705, user_id=4e7be52714b342beaefe916553ca7714, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:14.441380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:14.456605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1185/2331] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:14 2022] PUT /identity/v3/projects/b8f93a7f750346e195ae6543fda1d705/users/4e7be52714b342beaefe916553ca7714/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:14.463424 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:14.476112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc6ddb-4c63-49f8-ac63-f0e1f8e9cbc1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:14.554753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763a67f2-1151-4bb0-a3a3-de8933e13d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:14.555452 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763a67f2-1151-4bb0-a3a3-de8933e13d26 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.555736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763a67f2-1151-4bb0-a3a3-de8933e13d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.556026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763a67f2-1151-4bb0-a3a3-de8933e13d26 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.556766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763a67f2-1151-4bb0-a3a3-de8933e13d26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:14.558456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763a67f2-1151-4bb0-a3a3-de8933e13d26 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:14.570539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1147/2332] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:14 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:14.575831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:14.596104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856fd40-96b3-4c5f-97b3-2280a38740ed admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:14.605149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:14.606878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.607392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.607898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] PATH_INFO: `/v3/projects/b8f93a7f750346e195ae6543fda1d705/users/4e7be52714b342beaefe916553ca7714/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.611804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8f93a7f750346e195ae6543fda1d705, user_id=4e7be52714b342beaefe916553ca7714, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:14.617024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:14.636887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1186/2333] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:14 2022] PUT /identity/v3/projects/b8f93a7f750346e195ae6543fda1d705/users/4e7be52714b342beaefe916553ca7714/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:14.648815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d882f78-ad98-4444-a668-6deecd8775aa None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.649099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d882f78-ad98-4444-a668-6deecd8775aa None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.649378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d882f78-ad98-4444-a668-6deecd8775aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.689539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-6d882f78-ad98-4444-a668-6deecd8775aa None None] MFA Rules not processed for user `4e7be52714b342beaefe916553ca7714`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:14.691531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6d882f78-ad98-4444-a668-6deecd8775aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:14.801809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1148/2334] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:14 2022] POST /identity/v3/auth/tokens => generated 3295 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:14.824535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:14.827531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:14.933070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11d4fc8-000a-43a9-9889-e6e9e92e8a10 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:14.939483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:14.940707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:14.941231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:14.942287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:14.959932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:14.961441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:14.973858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1187/2335] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:14 2022] GET /identity/v3/auth/tokens => generated 3295 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:15.065132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d882f78-ad98-4444-a668-6deecd8775aa None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:15.076681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d882f78-ad98-4444-a668-6deecd8775aa None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:15.081296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91fa9c-b201-454a-97c9-f246c7e47759 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:15.082948 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91fa9c-b201-454a-97c9-f246c7e47759 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:15.083241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91fa9c-b201-454a-97c9-f246c7e47759 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:15.083532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91fa9c-b201-454a-97c9-f246c7e47759 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:15.117562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:15.155555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca91fa9c-b201-454a-97c9-f246c7e47759 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:15.157027 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca91fa9c-b201-454a-97c9-f246c7e47759 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:15.168039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1149/2336] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:15 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:18.068489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:18.150186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b82bdd-71a5-44fc-adea-982cd56cc64e service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:18.155327 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:18.156257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:18.156548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:18.156829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] PATH_INFO: `/v3/users/8bbb3a6e971043fdbf77a997743474f3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:18.158277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8bbb3a6e971043fdbf77a997743474f3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:18.160417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:18.172917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:18.237093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8bbb3a6e971043fdbf77a997743474f3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:18.248388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8bbb3a6e971043fdbf77a997743474f3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:18.249206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8bbb3a6e971043fdbf77a997743474f3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:18.259815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8bbb3a6e971043fdbf77a997743474f3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:18.280037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8bbb3a6e971043fdbf77a997743474f3'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:18.292515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1188/2337] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:18 2022] DELETE /identity/v3/users/8bbb3a6e971043fdbf77a997743474f3 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:18.299489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:18.300090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:18.300459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:18.330831 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:18.333332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:18.456492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1150/2338] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:18.467728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:18.469810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:18.557727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd20002-559f-4186-a023-f5bcbdc256d6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:18.562639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:18.563917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:18.563917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:18.564293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:18.575977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:18.577565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:18.589802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1189/2339] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:18.995389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:18.997160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:19.012095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b42aa37-d811-4e10-8764-bb99cd1f31af None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:19.018335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:19.019963 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:19.020253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:19.020538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] PATH_INFO: `/v3/projects/2ff621396c154c4f8767b25311b1c848` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:19.021929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ff621396c154c4f8767b25311b1c848)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:19.023951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:19.085005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ff621396c154c4f8767b25311b1c848', 'request_id': 'req-4795aa1b-b934-439d-b7f2-e30253c18864'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.094921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ff621396c154c4f8767b25311b1c848', 'request_id': 'req-4795aa1b-b934-439d-b7f2-e30253c18864'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.106865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ff621396c154c4f8767b25311b1c848', 'request_id': 'req-4795aa1b-b934-439d-b7f2-e30253c18864'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.120388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] The token cache is being invalidate because project 2ff621396c154c4f8767b25311b1c848 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:19.120726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.121357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.123010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1151/2340] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:18 2022] DELETE /identity/v3/projects/2ff621396c154c4f8767b25311b1c848 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:19.127543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:19.138645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc1c54c-c672-456a-9545-c30d307beb63 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:19.204469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:19.205371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:19.205653 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:19.205931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] PATH_INFO: `/v3/users/4e7be52714b342beaefe916553ca7714` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:19.207848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e7be52714b342beaefe916553ca7714)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:19.209822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:19.255451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:19.276766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e7be52714b342beaefe916553ca7714'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.288359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e7be52714b342beaefe916553ca7714'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.289026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e7be52714b342beaefe916553ca7714'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.300346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e7be52714b342beaefe916553ca7714'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.321214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-ce26ceb6-f0b4-4a48-a252-11c85700ae06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e7be52714b342beaefe916553ca7714'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.328985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1190/2341] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:19 2022] DELETE /identity/v3/users/4e7be52714b342beaefe916553ca7714 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:19.659288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:19.662577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:19.691225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795aa1b-b934-439d-b7f2-e30253c18864 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:19.770867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:19.790641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:19.792124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:19.792615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:19.793101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] PATH_INFO: `/v3/projects/b8f93a7f750346e195ae6543fda1d705` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:19.795447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8f93a7f750346e195ae6543fda1d705)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:19.801658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:19.883884 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8f93a7f750346e195ae6543fda1d705', 'request_id': 'req-7feedbff-bef9-41e6-9ef2-61b3efe041f8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.892017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8f93a7f750346e195ae6543fda1d705', 'request_id': 'req-7feedbff-bef9-41e6-9ef2-61b3efe041f8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.902531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8f93a7f750346e195ae6543fda1d705', 'request_id': 'req-7feedbff-bef9-41e6-9ef2-61b3efe041f8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.916104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] The token cache is being invalidate because project b8f93a7f750346e195ae6543fda1d705 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:19.916391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.917090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7feedbff-bef9-41e6-9ef2-61b3efe041f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:19.918753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1152/2342] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:19 2022] DELETE /identity/v3/projects/b8f93a7f750346e195ae6543fda1d705 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:19.926630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c9f02c-a029-47e3-adb1-50ab78e4af2e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:19.927153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c9f02c-a029-47e3-adb1-50ab78e4af2e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:19.927657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c9f02c-a029-47e3-adb1-50ab78e4af2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:19.960117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-35c9f02c-a029-47e3-adb1-50ab78e4af2e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:19.962337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-35c9f02c-a029-47e3-adb1-50ab78e4af2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:20.044887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1191/2343] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:20.054123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b310deb-e5d6-4713-a069-93770ff0a142 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.054529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b310deb-e5d6-4713-a069-93770ff0a142 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.054935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b310deb-e5d6-4713-a069-93770ff0a142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.075275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3b310deb-e5d6-4713-a069-93770ff0a142 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:20.076662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3b310deb-e5d6-4713-a069-93770ff0a142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:20.085373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1153/2344] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:20.090171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c9f02c-a029-47e3-adb1-50ab78e4af2e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.101239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c9f02c-a029-47e3-adb1-50ab78e4af2e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.106374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.107098 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.107388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.107670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.108430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:46:20.108742 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.110457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.125457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1192/2345] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:46:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:46:20.131373 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b310deb-e5d6-4713-a069-93770ff0a142 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.142043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b310deb-e5d6-4713-a069-93770ff0a142 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.147368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.147982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.148273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.148557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.149285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.151304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.188405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1154/2346] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:20 2022] POST /identity/v3/projects => generated 381 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:20.193311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.204859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac420ca-d0a9-4a78-a080-d5624f26bf1e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.276656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.277360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.277647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.278074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.278997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.281092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.306470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1193/2347] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:20 2022] POST /identity/v3/users => generated 291 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:20.312614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.331609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5603d8-9a69-4534-aea2-185328065773 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.340160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.341338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.341843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.342336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.343677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.346743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.370845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1155/2348] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:20 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:20.373554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.386113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2181da9-338c-4796-86b1-590624e9b237 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.391365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d84d1-5238-43aa-a57e-132581aea397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.392074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d84d1-5238-43aa-a57e-132581aea397 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.392364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d84d1-5238-43aa-a57e-132581aea397 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.392648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d84d1-5238-43aa-a57e-132581aea397 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.393407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d84d1-5238-43aa-a57e-132581aea397 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.396507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d84d1-5238-43aa-a57e-132581aea397 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.408400 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1194/2349] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:20 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:20.413008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.424421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b43da5-ac3f-42f1-99b7-3b99e4dd4f08 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.429999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.430946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.431232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.431512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] PATH_INFO: `/v3/projects/0edfc73bd5e8480197e8d6ac2b699be1/users/62ff6558475e44f18273ca8e1a4a274c/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.451130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0edfc73bd5e8480197e8d6ac2b699be1, user_id=62ff6558475e44f18273ca8e1a4a274c, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.455221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.471964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1156/2350] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:20 2022] PUT /identity/v3/projects/0edfc73bd5e8480197e8d6ac2b699be1/users/62ff6558475e44f18273ca8e1a4a274c/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:20.485013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ef4828-d8f3-4751-b3ad-984ad1d0787c None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.485508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ef4828-d8f3-4751-b3ad-984ad1d0787c None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.486026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ef4828-d8f3-4751-b3ad-984ad1d0787c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.536074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-69ef4828-d8f3-4751-b3ad-984ad1d0787c None None] MFA Rules not processed for user `62ff6558475e44f18273ca8e1a4a274c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:20.538259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-69ef4828-d8f3-4751-b3ad-984ad1d0787c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:20.545770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:20.649880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1195/2351] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:20 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:20.655383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.732727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323dad13-911d-4203-bc6d-424112be0f49 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.738131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.739356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.739643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.739922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.740636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.742648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.772981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1157/2352] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:20 2022] POST /identity/v3/projects => generated 379 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:20.778807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ef4828-d8f3-4751-b3ad-984ad1d0787c None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.791751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ef4828-d8f3-4751-b3ad-984ad1d0787c None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.857517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.858240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.858552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.858833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.859565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.861531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.885320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1196/2353] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:20 2022] POST /identity/v3/users => generated 296 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:20.889921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.896836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:20.906623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c1a837-798d-4fb4-a7c7-569cec4d4621 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.914825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.915621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.915908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.916243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.917014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.919143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:20.943362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1158/2354] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:20 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:20.945268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:20.964547 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a34634-7a8c-4abf-b438-b21d8fff445e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:20.972807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:20.973970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:20.974469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:20.974925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] PATH_INFO: `/v3/projects/e055b1de55dd44b48f6c1c32d56a02e5/users/be24118d22f64eabaaa9444b28658bc9/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:20.995237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e055b1de55dd44b48f6c1c32d56a02e5, user_id=be24118d22f64eabaaa9444b28658bc9, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:20.998051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:21.014650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1197/2355] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:20 2022] PUT /identity/v3/projects/e055b1de55dd44b48f6c1c32d56a02e5/users/be24118d22f64eabaaa9444b28658bc9/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:21.020356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:21.038846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f4296b-ca52-4e3b-9254-a577d3b2acd9 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:21.121908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a442b7f-58ae-4ad8-97ad-23b49191a6f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:21.122871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a442b7f-58ae-4ad8-97ad-23b49191a6f5 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:21.123311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a442b7f-58ae-4ad8-97ad-23b49191a6f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:21.123691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a442b7f-58ae-4ad8-97ad-23b49191a6f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:21.124678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a442b7f-58ae-4ad8-97ad-23b49191a6f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:21.126832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a442b7f-58ae-4ad8-97ad-23b49191a6f5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:21.139222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1159/2356] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:21 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:21.143849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:21.155304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e891d27c-6b6b-43d9-9519-83f406177106 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:21.160563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:21.161486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:21.161889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:21.162241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] PATH_INFO: `/v3/projects/e055b1de55dd44b48f6c1c32d56a02e5/users/be24118d22f64eabaaa9444b28658bc9/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:21.164554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e055b1de55dd44b48f6c1c32d56a02e5, user_id=be24118d22f64eabaaa9444b28658bc9, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:21.167437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:21.180492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1198/2357] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:21 2022] PUT /identity/v3/projects/e055b1de55dd44b48f6c1c32d56a02e5/users/be24118d22f64eabaaa9444b28658bc9/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:21.192996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4cc1a3-a215-4ce9-b739-e71cea48aea1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:21.193265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4cc1a3-a215-4ce9-b739-e71cea48aea1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:21.193554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4cc1a3-a215-4ce9-b739-e71cea48aea1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:21.225508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9f4cc1a3-a215-4ce9-b739-e71cea48aea1 None None] MFA Rules not processed for user `be24118d22f64eabaaa9444b28658bc9`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:21.226912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9f4cc1a3-a215-4ce9-b739-e71cea48aea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:21.312094 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1160/2358] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:21 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:21.325824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:21.327684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:21.394030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03206ab-c988-4802-8d03-314002f564b9 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:21.397999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:21.398700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:21.398987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:21.399269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:21.409769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:21.411414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:21.417274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:21.423990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1199/2359] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:21 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:21.817243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4cc1a3-a215-4ce9-b739-e71cea48aea1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:21.827527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4cc1a3-a215-4ce9-b739-e71cea48aea1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:21.832746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'be24118d22f64eabaaa9444b28658bc9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e055b1de55dd44b48f6c1c32d56a02e5', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:21.833435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:21.833724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:21.834023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:21.834810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:46:21.835130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:21.836791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:21.851005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1161/2360] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:46:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:46:21.857504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:21.868819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571d479-4de2-4e2f-8ab9-d43ba6bf1b67 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:21.874487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'be24118d22f64eabaaa9444b28658bc9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e055b1de55dd44b48f6c1c32d56a02e5', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:21.875201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:21.875490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:21.875774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:21.876489 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:21.878535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:21.914505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1200/2361] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:21 2022] POST /identity/v3/projects => generated 353 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:21.918891 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:21.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ec2a0-68af-474e-8acb-75713907a027 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:22.001223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'be24118d22f64eabaaa9444b28658bc9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e055b1de55dd44b48f6c1c32d56a02e5', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:22.001972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:22.002188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:22.002500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:22.003269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:46:22.003615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:22.005836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:22.007937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:22.020745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1162/2362] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:46:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:46:22.069627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:22.081290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f7bb3-74b1-4554-b58f-17502151550c tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:22.086944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'be24118d22f64eabaaa9444b28658bc9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e055b1de55dd44b48f6c1c32d56a02e5', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:22.087842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:22.088132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:22.088613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] PATH_INFO: `/v3/projects/be07c4ae2f7d4f45ac9589e935baec8d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:22.090032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorizing `identity:delete_project(project_id=be07c4ae2f7d4f45ac9589e935baec8d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:22.092040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:22.148569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be07c4ae2f7d4f45ac9589e935baec8d', 'request_id': 'req-0afabf45-9961-43a6-93a5-b47f269b657d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.158082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be07c4ae2f7d4f45ac9589e935baec8d', 'request_id': 'req-0afabf45-9961-43a6-93a5-b47f269b657d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.165911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be07c4ae2f7d4f45ac9589e935baec8d', 'request_id': 'req-0afabf45-9961-43a6-93a5-b47f269b657d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.179821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] The token cache is being invalidate because project be07c4ae2f7d4f45ac9589e935baec8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:22.180316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.181426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0afabf45-9961-43a6-93a5-b47f269b657d tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.183213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1201/2363] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:22 2022] DELETE /identity/v3/projects/be07c4ae2f7d4f45ac9589e935baec8d => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:22.464152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:22.466532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:22.543154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e0cc6d-249b-42ca-9aae-01bd2d0c4bc0 tempest-VolumeQuotaClassesTest-446328807 tempest-VolumeQuotaClassesTest-446328807-project-admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:22.609758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:22.610987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:22.611237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:22.611554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] PATH_INFO: `/v3/users/62ff6558475e44f18273ca8e1a4a274c` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:22.612957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=62ff6558475e44f18273ca8e1a4a274c)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:22.615546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:22.679715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '62ff6558475e44f18273ca8e1a4a274c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.689647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '62ff6558475e44f18273ca8e1a4a274c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.699378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '62ff6558475e44f18273ca8e1a4a274c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.700165 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '62ff6558475e44f18273ca8e1a4a274c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.721441 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '62ff6558475e44f18273ca8e1a4a274c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:22.735178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1163/2364] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:22 2022] DELETE /identity/v3/users/62ff6558475e44f18273ca8e1a4a274c => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:22.741414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:22.742122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:22.742297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:22.774418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:22.775962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:22.850993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1202/2365] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:22.861726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:22.864375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:22.893151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:22.943312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b4fd01-b6c3-49e2-86ea-4a6d5b9ed7d3 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:22.947524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:22.948258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:22.948589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:22.948930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:22.962358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:22.963921 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:22.975321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1164/2366] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:23.506619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:23.508405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:23.524134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a30c67-0744-44db-8e58-51f5df439f45 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:23.527481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:23.528374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:23.528658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:23.528937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] PATH_INFO: `/v3/projects/0edfc73bd5e8480197e8d6ac2b699be1` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:23.530217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0edfc73bd5e8480197e8d6ac2b699be1)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:23.532257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:23.600282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0edfc73bd5e8480197e8d6ac2b699be1', 'request_id': 'req-f06c546b-f661-4d15-854f-7815026f5b70'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.614840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0edfc73bd5e8480197e8d6ac2b699be1', 'request_id': 'req-f06c546b-f661-4d15-854f-7815026f5b70'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.616683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=85,keystone:INSERT=3,keystone:DELETE=12 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:23.625401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0edfc73bd5e8480197e8d6ac2b699be1', 'request_id': 'req-f06c546b-f661-4d15-854f-7815026f5b70'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.646063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] The token cache is being invalidate because project 0edfc73bd5e8480197e8d6ac2b699be1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:23.646359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.647115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.648759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1203/2367] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:23 2022] DELETE /identity/v3/projects/0edfc73bd5e8480197e8d6ac2b699be1 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:23.653396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:23.665521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81440d19-2c5a-43e7-95ff-fa172cff9ec1 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:23.772528 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:23.774306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:23.775038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:23.775535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] PATH_INFO: `/v3/users/be24118d22f64eabaaa9444b28658bc9` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:23.779568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] RBAC: Authorizing `identity:delete_user(user_id=be24118d22f64eabaaa9444b28658bc9)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:23.783587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:23.913933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be24118d22f64eabaaa9444b28658bc9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.957795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be24118d22f64eabaaa9444b28658bc9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.984384 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be24118d22f64eabaaa9444b28658bc9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:23.985744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be24118d22f64eabaaa9444b28658bc9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:24.023278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b3f7cf6b-80c9-40e2-a56e-72e88f90b03c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be24118d22f64eabaaa9444b28658bc9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:24.038313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1165/2368] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:23 2022] DELETE /identity/v3/users/be24118d22f64eabaaa9444b28658bc9 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:24.936189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:24.938655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:24.965020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06c546b-f661-4d15-854f-7815026f5b70 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:25.085121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:25.086890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.087784 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.088534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] PATH_INFO: `/v3/projects/e055b1de55dd44b48f6c1c32d56a02e5` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.092152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e055b1de55dd44b48f6c1c32d56a02e5)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:25.096131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:25.157991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e055b1de55dd44b48f6c1c32d56a02e5', 'request_id': 'req-f95e287c-2851-4198-9570-e2e0e3e4884c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:25.170264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e055b1de55dd44b48f6c1c32d56a02e5', 'request_id': 'req-f95e287c-2851-4198-9570-e2e0e3e4884c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:25.179609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e055b1de55dd44b48f6c1c32d56a02e5', 'request_id': 'req-f95e287c-2851-4198-9570-e2e0e3e4884c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:25.196335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] The token cache is being invalidate because project e055b1de55dd44b48f6c1c32d56a02e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:25.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:25.197560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f95e287c-2851-4198-9570-e2e0e3e4884c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:25.199508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1204/2369] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:24 2022] DELETE /identity/v3/projects/e055b1de55dd44b48f6c1c32d56a02e5 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:25.205917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b509d-ff29-4ae8-9b14-f0d59904841f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.208162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b509d-ff29-4ae8-9b14-f0d59904841f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.208541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b509d-ff29-4ae8-9b14-f0d59904841f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.241440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-065b509d-ff29-4ae8-9b14-f0d59904841f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:25.243820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-065b509d-ff29-4ae8-9b14-f0d59904841f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:25.325897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:25.369749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1166/2370] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:25.372757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fe886f-dd89-4600-b326-bf1299865df8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.373141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fe886f-dd89-4600-b326-bf1299865df8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.373643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fe886f-dd89-4600-b326-bf1299865df8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.411507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-09fe886f-dd89-4600-b326-bf1299865df8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:25.413747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-09fe886f-dd89-4600-b326-bf1299865df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:25.436834 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1205/2371] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:25.443804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065b509d-ff29-4ae8-9b14-f0d59904841f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:25.457823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065b509d-ff29-4ae8-9b14-f0d59904841f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:25.464096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:25.464899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.465215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.465688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.467332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:46:25.467779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:25.469627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:25.487691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1167/2372] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:46:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:46:25.494511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fe886f-dd89-4600-b326-bf1299865df8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:25.512187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fe886f-dd89-4600-b326-bf1299865df8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:25.521116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:25.522275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.522804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.523295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.524623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:25.528592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:25.580904 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1206/2373] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:25 2022] POST /identity/v3/projects => generated 385 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:25.586039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:25.602089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b6af1-0f00-452e-9d10-81077f94e5db admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:25.708607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:25.709218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.709504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.709783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.710609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:25.712623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:25.747766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1168/2374] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:25 2022] POST /identity/v3/users => generated 293 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:25.749363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:25.763362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52146dea-759a-41a8-a685-cf70219b1f4e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:25.770125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:25.771084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.771444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.771779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.772690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:25.774984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:25.793480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1207/2375] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:25 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:25.799864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:25.818389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baefa9f-3be7-4a22-a2ea-bdbc9707cbc7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:25.826287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44026dfa-f414-42ad-a987-94cbc96d724c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:25.827330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44026dfa-f414-42ad-a987-94cbc96d724c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.827739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44026dfa-f414-42ad-a987-94cbc96d724c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.828148 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44026dfa-f414-42ad-a987-94cbc96d724c admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.829407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44026dfa-f414-42ad-a987-94cbc96d724c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:25.833080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44026dfa-f414-42ad-a987-94cbc96d724c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:25.852178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1169/2376] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:25 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:25.857345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:25.868481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f90f44e-6c9b-42c2-aa0a-8a3f18422be4 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:25.873997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:25.875332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.875818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.876217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] PATH_INFO: `/v3/projects/6f453543d313450e9136ce36938df197/users/eed4ee8b1beb4e34b51b7ef5d0d811b7/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.897435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f453543d313450e9136ce36938df197, user_id=eed4ee8b1beb4e34b51b7ef5d0d811b7, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:25.900559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:25.920356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1208/2377] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:25 2022] PUT /identity/v3/projects/6f453543d313450e9136ce36938df197/users/eed4ee8b1beb4e34b51b7ef5d0d811b7/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:25.932438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef21ce7-2cba-4393-b320-7afd73945fa7 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:25.933087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef21ce7-2cba-4393-b320-7afd73945fa7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:25.933656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef21ce7-2cba-4393-b320-7afd73945fa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:25.971867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7ef21ce7-2cba-4393-b320-7afd73945fa7 None None] MFA Rules not processed for user `eed4ee8b1beb4e34b51b7ef5d0d811b7`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:25.973123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7ef21ce7-2cba-4393-b320-7afd73945fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:26.077780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1170/2378] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:25 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:26.084838 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:26.108141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=9,keystone:SELECT=88,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:26.217310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5299b7-2f0a-40ff-b7ab-334b8cfe4faa admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:26.225912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:26.227158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:26.228021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:26.228274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:26.229617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:26.233735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:26.289569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1209/2379] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:26 2022] POST /identity/v3/projects => generated 387 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:26.290690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef21ce7-2cba-4393-b320-7afd73945fa7 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:26.312043 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef21ce7-2cba-4393-b320-7afd73945fa7 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:26.387958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:26.432556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:26.433468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:26.433860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:26.434249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:26.435130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:26.437494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:26.495584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:26.502665 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1171/2380] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:26 2022] POST /identity/v3/users => generated 294 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:26.520003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748bea4f-1f7b-4d7f-8bbd-ef63bb9ba020 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:26.531528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:26.532201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:26.532371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:26.532630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:26.533827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:26.535893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:26.564046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1210/2381] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:26 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:26.568309 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:26.579926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cbbf76-63f7-4a9d-94d0-320385f6a1ca admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:26.586344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f680fe5-d61e-492e-ac0d-034ca5e753d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:26.587183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f680fe5-d61e-492e-ac0d-034ca5e753d6 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:26.587540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f680fe5-d61e-492e-ac0d-034ca5e753d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:26.587938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f680fe5-d61e-492e-ac0d-034ca5e753d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:26.588659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f680fe5-d61e-492e-ac0d-034ca5e753d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:26.590614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f680fe5-d61e-492e-ac0d-034ca5e753d6 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:26.608585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:26.609112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1172/2382] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:26 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:26.624616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf2bf68-1548-41e2-a1d6-a971bba2c84c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:26.644575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:26.645662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:26.646081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:26.646505 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] PATH_INFO: `/v3/projects/07a90e6bc9d1418088705b4d9997aca7/users/44fd82d28e4648fe9c92b3cde57e949c/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:26.681434 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07a90e6bc9d1418088705b4d9997aca7, user_id=44fd82d28e4648fe9c92b3cde57e949c, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:26.685286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:26.709734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1211/2383] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:26 2022] PUT /identity/v3/projects/07a90e6bc9d1418088705b4d9997aca7/users/44fd82d28e4648fe9c92b3cde57e949c/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:26.721476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a23c2-2303-4ec1-b59d-03afb8838bf4 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:26.721798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a23c2-2303-4ec1-b59d-03afb8838bf4 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:26.722072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a23c2-2303-4ec1-b59d-03afb8838bf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:26.758142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ef8a23c2-2303-4ec1-b59d-03afb8838bf4 None None] MFA Rules not processed for user `44fd82d28e4648fe9c92b3cde57e949c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:26.759512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ef8a23c2-2303-4ec1-b59d-03afb8838bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:26.918703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1173/2384] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:26.919999 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:27.066099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8188d548-27d7-4cfa-982b-d9af92462cb5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:27.076084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:27.076703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.077559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.077692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.079083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:27.084403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:27.135514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1212/2385] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:26 2022] POST /identity/v3/projects => generated 385 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:27.140031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8a23c2-2303-4ec1-b59d-03afb8838bf4 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:27.152816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8a23c2-2303-4ec1-b59d-03afb8838bf4 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:27.242317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:27.243265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.243503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.243798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.244511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:27.246541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:27.275256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1174/2386] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:27 2022] POST /identity/v3/users => generated 299 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:46:27.279962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:27.292078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e8393d-b74e-4b29-9f81-ffacc0659e77 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:27.297474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:27.298174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.298525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.298770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.299622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:27.301850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:27.314312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1213/2387] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:27.318912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:27.330104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f5253-218c-4ff3-9cb2-d3e590075001 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:27.335713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:27.336643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.336951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.337209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] PATH_INFO: `/v3/projects/a1cb5f85b5234caa8abcc8b2a65c5fe1/users/bdea24691b384f0f946d8527f1d87bcc/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.356536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1cb5f85b5234caa8abcc8b2a65c5fe1, user_id=bdea24691b384f0f946d8527f1d87bcc, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:27.360691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:27.377519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1175/2388] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:27 2022] PUT /identity/v3/projects/a1cb5f85b5234caa8abcc8b2a65c5fe1/users/bdea24691b384f0f946d8527f1d87bcc/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:27.381947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:27.395394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bee13c-cc53-494c-bb8b-a0b5782dcb0f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:27.455940 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:27.520818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584ec847-5a33-4e71-9c25-12b6f00c7b7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:27.521993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ec847-5a33-4e71-9c25-12b6f00c7b7a admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.522600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ec847-5a33-4e71-9c25-12b6f00c7b7a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.523336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ec847-5a33-4e71-9c25-12b6f00c7b7a admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.524842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584ec847-5a33-4e71-9c25-12b6f00c7b7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:27.536694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584ec847-5a33-4e71-9c25-12b6f00c7b7a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:27.560472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1214/2389] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:27.564786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:27.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eee95b-91c1-422c-9a06-ba9805097d47 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:27.591176 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:27.592382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.592686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.592979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] PATH_INFO: `/v3/projects/a1cb5f85b5234caa8abcc8b2a65c5fe1/users/bdea24691b384f0f946d8527f1d87bcc/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.596140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1cb5f85b5234caa8abcc8b2a65c5fe1, user_id=bdea24691b384f0f946d8527f1d87bcc, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:27.599167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:27.606458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:27.615636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1176/2390] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:27 2022] PUT /identity/v3/projects/a1cb5f85b5234caa8abcc8b2a65c5fe1/users/bdea24691b384f0f946d8527f1d87bcc/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:27.626718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce217ee-a48e-4d34-bacb-0aa2780060d5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.627180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce217ee-a48e-4d34-bacb-0aa2780060d5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.627432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce217ee-a48e-4d34-bacb-0aa2780060d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.675167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5ce217ee-a48e-4d34-bacb-0aa2780060d5 None None] MFA Rules not processed for user `bdea24691b384f0f946d8527f1d87bcc`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:27.675167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5ce217ee-a48e-4d34-bacb-0aa2780060d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:27.806743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1215/2391] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:27 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:27.822542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:27.825306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:27.908455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4811b-e13f-4151-a815-7214462d5eda admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:27.914231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:27.915099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:27.916302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:27.916726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:27.930125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:27.932710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:27.948252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1177/2392] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:27 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:28.818127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79e4038-85e7-4518-9022-82c1d1c862d4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:28.819089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79e4038-85e7-4518-9022-82c1d1c862d4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:28.819089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79e4038-85e7-4518-9022-82c1d1c862d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:28.852208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d79e4038-85e7-4518-9022-82c1d1c862d4 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:28.853812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d79e4038-85e7-4518-9022-82c1d1c862d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:28.935310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1216/2393] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:46:28 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:29.006210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:29.009154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:29.111937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee39237-c3eb-4033-9ffa-b50ecdbe3fb1 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:29.118663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:29.119524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:29.120038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:29.120320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:29.127753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79e4038-85e7-4518-9022-82c1d1c862d4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:29.130845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:29.132520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:29.145453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79e4038-85e7-4518-9022-82c1d1c862d4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:29.146154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1178/2394] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:46:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:46:29.155781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9dfa1f-e15c-43d2-bf2f-d176d79def9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:29.156836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9dfa1f-e15c-43d2-bf2f-d176d79def9c service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:29.157312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9dfa1f-e15c-43d2-bf2f-d176d79def9c service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:29.157781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9dfa1f-e15c-43d2-bf2f-d176d79def9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:29.274771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9dfa1f-e15c-43d2-bf2f-d176d79def9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:29.276438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9dfa1f-e15c-43d2-bf2f-d176d79def9c service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:29.282488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:29.290158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1217/2395] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:34.303327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:34.317509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9c750-a0db-454a-ac0f-3fab40057a2e service swift] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:34.325046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:34.327431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:34.328376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:34.329258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:34.439778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:34.442351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:34.458269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1179/2396] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:34 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:35.262125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd2e8c-0234-4eb2-a685-3ffe1fc84cfd None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:35.263333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd2e8c-0234-4eb2-a685-3ffe1fc84cfd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:35.263846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd2e8c-0234-4eb2-a685-3ffe1fc84cfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:35.315254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-3bfd2e8c-0234-4eb2-a685-3ffe1fc84cfd None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:35.318457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3bfd2e8c-0234-4eb2-a685-3ffe1fc84cfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:35.332831 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1218/2397] 10.222.0.72 () {58 vars in 967 bytes} [Mon Jul 11 14:46:35 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:35.374733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:35.406118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ab4f4f-5270-4477-86e7-5a92be91ad34 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:35.424515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffecbf25-2215-4f6d-8d91-58b428c6adcd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:35.427281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffecbf25-2215-4f6d-8d91-58b428c6adcd service swift] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:35.428300 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffecbf25-2215-4f6d-8d91-58b428c6adcd service swift] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:35.429504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffecbf25-2215-4f6d-8d91-58b428c6adcd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:35.459337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffecbf25-2215-4f6d-8d91-58b428c6adcd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:35.460157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffecbf25-2215-4f6d-8d91-58b428c6adcd service swift] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:35.484878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1180/2398] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:46:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:46:37.841391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd2e8c-0234-4eb2-a685-3ffe1fc84cfd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:37.999926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd2e8c-0234-4eb2-a685-3ffe1fc84cfd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:38.011371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:38.013311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:38.013901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:38.014477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] PATH_INFO: `/v3/users/eed4ee8b1beb4e34b51b7ef5d0d811b7` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:38.017307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eed4ee8b1beb4e34b51b7ef5d0d811b7)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:38.022215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:38.195563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eed4ee8b1beb4e34b51b7ef5d0d811b7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:38.216484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eed4ee8b1beb4e34b51b7ef5d0d811b7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:38.217658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eed4ee8b1beb4e34b51b7ef5d0d811b7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:38.246767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eed4ee8b1beb4e34b51b7ef5d0d811b7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:38.282672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eed4ee8b1beb4e34b51b7ef5d0d811b7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:38.298477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1219/2399] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:37 2022] DELETE /identity/v3/users/eed4ee8b1beb4e34b51b7ef5d0d811b7 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:38.303730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:38.304674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:38.305118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:38.378113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:38.384490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:38.403272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:38.521650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1181/2400] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:38.534918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:38.536635 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:38.626719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b188ac-049e-421e-8318-ddcf8566b8d5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:38.638114 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:38.639061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:38.639475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:38.639933 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:38.659166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:38.661637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:38.678131 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1220/2401] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:39.184357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:39.188109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:39.239780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85379552-4df1-4140-9bcf-335c9980e35c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:39.263888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:39.268255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:39.270289 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:39.271783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] PATH_INFO: `/v3/projects/6f453543d313450e9136ce36938df197` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:39.283033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f453543d313450e9136ce36938df197)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:39.287558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:39.497303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f453543d313450e9136ce36938df197', 'request_id': 'req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:39.524105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f453543d313450e9136ce36938df197', 'request_id': 'req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:39.566722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f453543d313450e9136ce36938df197', 'request_id': 'req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:39.598970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] The token cache is being invalidate because project 6f453543d313450e9136ce36938df197 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:39.599686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:39.601372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:39.605159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1182/2402] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:39 2022] DELETE /identity/v3/projects/6f453543d313450e9136ce36938df197 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:39.610832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:39.644966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b39d1e5-c64a-4bb2-97fe-a6c7a864c506 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:39.680909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:39.817272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:39.819024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:39.819585 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:39.820184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] PATH_INFO: `/v3/users/44fd82d28e4648fe9c92b3cde57e949c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:39.823055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44fd82d28e4648fe9c92b3cde57e949c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:39.849816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:39.977526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44fd82d28e4648fe9c92b3cde57e949c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:39.988895 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44fd82d28e4648fe9c92b3cde57e949c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:39.989728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44fd82d28e4648fe9c92b3cde57e949c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:40.002891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44fd82d28e4648fe9c92b3cde57e949c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:40.025763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44fd82d28e4648fe9c92b3cde57e949c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:40.036432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1221/2403] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:39 2022] DELETE /identity/v3/users/44fd82d28e4648fe9c92b3cde57e949c => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:40.783698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:40.795516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:40.823461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08de021a-1e2e-4492-9bd5-0ad2ec50ef3d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:41.015773 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:41.016758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:41.017089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:41.017397 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] PATH_INFO: `/v3/projects/07a90e6bc9d1418088705b4d9997aca7` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:41.019965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=07a90e6bc9d1418088705b4d9997aca7)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:41.036354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:41.190340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:41.208621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07a90e6bc9d1418088705b4d9997aca7', 'request_id': 'req-508f163c-97ec-423b-8df6-d20da313a1aa'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.221387 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07a90e6bc9d1418088705b4d9997aca7', 'request_id': 'req-508f163c-97ec-423b-8df6-d20da313a1aa'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.248924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07a90e6bc9d1418088705b4d9997aca7', 'request_id': 'req-508f163c-97ec-423b-8df6-d20da313a1aa'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.290017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] The token cache is being invalidate because project 07a90e6bc9d1418088705b4d9997aca7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:41.290322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.291329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.294685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1183/2404] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:40 2022] DELETE /identity/v3/projects/07a90e6bc9d1418088705b4d9997aca7 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:41.299897 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:41.302080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:41.340795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63b7bad-7ced-45f7-bd95-b32222826d81 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:41.575115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:41.576687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:41.577124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:41.577549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] PATH_INFO: `/v3/users/bdea24691b384f0f946d8527f1d87bcc` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:41.591787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bdea24691b384f0f946d8527f1d87bcc)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:41.595352 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:41.731514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:41.767356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bdea24691b384f0f946d8527f1d87bcc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.790637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bdea24691b384f0f946d8527f1d87bcc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.791674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bdea24691b384f0f946d8527f1d87bcc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.818418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bdea24691b384f0f946d8527f1d87bcc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.860174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cf889637-733b-44e1-a5ba-d07cb710c9b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bdea24691b384f0f946d8527f1d87bcc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:41.881969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1222/2405] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:41 2022] DELETE /identity/v3/users/bdea24691b384f0f946d8527f1d87bcc => generated 0 bytes in 582 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:42.669682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:42.671344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:42.711804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508f163c-97ec-423b-8df6-d20da313a1aa admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:42.885600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:42.886994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:42.887648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:42.887955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] PATH_INFO: `/v3/projects/a1cb5f85b5234caa8abcc8b2a65c5fe1` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:42.892389 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1cb5f85b5234caa8abcc8b2a65c5fe1)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:42.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:43.026691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1cb5f85b5234caa8abcc8b2a65c5fe1', 'request_id': 'req-84272d74-2990-462d-937b-a068b9c25e96'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:43.049209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1cb5f85b5234caa8abcc8b2a65c5fe1', 'request_id': 'req-84272d74-2990-462d-937b-a068b9c25e96'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:43.068541 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1cb5f85b5234caa8abcc8b2a65c5fe1', 'request_id': 'req-84272d74-2990-462d-937b-a068b9c25e96'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:43.088281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] The token cache is being invalidate because project a1cb5f85b5234caa8abcc8b2a65c5fe1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:43.088826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:43.090137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-84272d74-2990-462d-937b-a068b9c25e96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:43.121984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1184/2406] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:42 2022] DELETE /identity/v3/projects/a1cb5f85b5234caa8abcc8b2a65c5fe1 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:43.140413 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538c14d7-138c-440f-8895-3c5fbbf44793 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:43.156115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538c14d7-138c-440f-8895-3c5fbbf44793 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:43.157354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538c14d7-138c-440f-8895-3c5fbbf44793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:43.180435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-538c14d7-138c-440f-8895-3c5fbbf44793 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:43.182174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-538c14d7-138c-440f-8895-3c5fbbf44793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:43.290859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1223/2407] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:43.301156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d529f77a-ace0-434b-acd5-3b0d88ed9c51 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:43.301518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d529f77a-ace0-434b-acd5-3b0d88ed9c51 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:43.301718 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d529f77a-ace0-434b-acd5-3b0d88ed9c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:43.335512 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-d529f77a-ace0-434b-acd5-3b0d88ed9c51 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:43.336902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d529f77a-ace0-434b-acd5-3b0d88ed9c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:43.352994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538c14d7-138c-440f-8895-3c5fbbf44793 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:43.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538c14d7-138c-440f-8895-3c5fbbf44793 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:43.364585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1185/2408] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:46:43.371096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:43.371797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:43.372080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:43.372359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:43.373115 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:46:43.373428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:43.376039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:43.399883 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1224/2409] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:46:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:46:43.402527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d529f77a-ace0-434b-acd5-3b0d88ed9c51 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:43.413573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d529f77a-ace0-434b-acd5-3b0d88ed9c51 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:43.422981 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:43.423678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:43.423959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:43.424238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:43.425070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:43.427486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:43.489007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1186/2410] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:46:43 2022] POST /identity/v3/projects => generated 369 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:43.491432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:43.510116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb828e67-32c2-4f94-a6b7-bb845b8e8222 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:43.698550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:43.700136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:43.700994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:43.701944 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:43.703373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:43.709259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:43.783159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1225/2411] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:46:43 2022] POST /identity/v3/users => generated 285 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:46:43.789313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:43.817024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0fc2d7-0403-4b7e-8fa1-57eeb5df4d36 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:43.851107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:43.857547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:43.857758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:43.858237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:43.860531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:43.873394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:43.933138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1187/2412] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:43 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:46:43.943844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:43.973447 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66fe389-d529-4e06-951c-7c8e9d08a0b1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:43.985213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f6dec-fc4c-4192-a383-42c5b1a69666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:43.986251 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f6dec-fc4c-4192-a383-42c5b1a69666 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:43.990362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f6dec-fc4c-4192-a383-42c5b1a69666 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:43.990362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f6dec-fc4c-4192-a383-42c5b1a69666 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:43.990362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f6dec-fc4c-4192-a383-42c5b1a69666 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:43.994643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f6dec-fc4c-4192-a383-42c5b1a69666 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:44.028580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1226/2413] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:46:43 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:46:44.032744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:44.058899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82191677-a52d-4f7f-82a6-12ff23ddfba3 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:44.074868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:44.077221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:44.078067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:44.078731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] PATH_INFO: `/v3/projects/0f3ad328ff6744e4bcef8d39a4324044/users/888e7afcc0834d55a8cee3fd6a4b8f44/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:44.116853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f3ad328ff6744e4bcef8d39a4324044, user_id=888e7afcc0834d55a8cee3fd6a4b8f44, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:44.122642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:44.130121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:44.163464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1188/2414] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:46:44 2022] PUT /identity/v3/projects/0f3ad328ff6744e4bcef8d39a4324044/users/888e7afcc0834d55a8cee3fd6a4b8f44/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:44.247376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:44.248135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:44.248135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:44.282752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] MFA Rules not processed for user `888e7afcc0834d55a8cee3fd6a4b8f44`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:44.284081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:44.304133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:44.400581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1227/2415] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:46:44 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:44.414007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:44.418305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:44.674759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea7ad38-4ab1-4b9e-a136-60409a59bf36 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:44.674759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fb6c2e2ddb3b4d488fa25fa8e258fec0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:44.674759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:44.674759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:44.675993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:44.675993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:44.675993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:44.675993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1189/2416] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:44 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:46:49.344601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:49.346415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:49.465105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c686b4-e80b-455d-93a7-6131405d87e5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:49.477250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:49.479326 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:49.490092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:49.490995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:49.491287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:49.491571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:49.497145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:49.523961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:49.527099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:49.545483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1228/2417] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:46:49 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:46:49.625994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49cc04d-7485-433a-987a-336c7f4186fa service glance] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:49.631766 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:49.632735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:49.633063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:49.633385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] PATH_INFO: `/v3/users/82025bd919ce404b96491a19262d4281` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:49.634982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82025bd919ce404b96491a19262d4281)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:49.637022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:49.702211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82025bd919ce404b96491a19262d4281'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:49.712538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82025bd919ce404b96491a19262d4281'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:49.724266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82025bd919ce404b96491a19262d4281'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:49.725703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82025bd919ce404b96491a19262d4281'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:49.750828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82025bd919ce404b96491a19262d4281'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:49.760551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1190/2418] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:49 2022] DELETE /identity/v3/users/82025bd919ce404b96491a19262d4281 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:50.118382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:50.148166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b50b219-50aa-48b6-8cdb-96ed8d7c9e9d service placement] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:50.267815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:50.269086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:50.269556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:50.270026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] PATH_INFO: `/v3/projects/3e77cfd4544c45468c57510de6034e38` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:50.272916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e77cfd4544c45468c57510de6034e38)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:50.276410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:50.348880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:50.366054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e77cfd4544c45468c57510de6034e38', 'request_id': 'req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.378872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e77cfd4544c45468c57510de6034e38', 'request_id': 'req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.389931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e77cfd4544c45468c57510de6034e38', 'request_id': 'req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.415024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] The token cache is being invalidate because project 3e77cfd4544c45468c57510de6034e38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:50.415495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.416561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.419344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1229/2419] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:50 2022] DELETE /identity/v3/projects/3e77cfd4544c45468c57510de6034e38 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:50.426142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:50.428323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:50.447534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23d136-2bec-4a08-931e-0c88d7a95972 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:50.465762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:50.545900 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:50.547881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:50.548574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:50.549036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] PATH_INFO: `/v3/users/8da898ceacd44be996652faccba3381a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:50.552137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] RBAC: Authorizing `identity:delete_user(user_id=8da898ceacd44be996652faccba3381a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:50.556231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:50.661507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8da898ceacd44be996652faccba3381a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.676362 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8da898ceacd44be996652faccba3381a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.691974 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8da898ceacd44be996652faccba3381a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.693201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8da898ceacd44be996652faccba3381a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.725934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9d5ea26e-2f6d-4cd7-a932-713c46b4210c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8da898ceacd44be996652faccba3381a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:50.738544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1191/2420] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:46:50 2022] DELETE /identity/v3/users/8da898ceacd44be996652faccba3381a => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:46:51.089847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:51.094020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:51.125762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc01ccd9-bee3-48a3-ba86-3ab92c685a1b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:51.255283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:51.257325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:51.257833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:51.258607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] PATH_INFO: `/v3/projects/0b70d99ff0094ac6b018c6f8a907e31b` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:51.260411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b70d99ff0094ac6b018c6f8a907e31b)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:51.262863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:51.347950 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b70d99ff0094ac6b018c6f8a907e31b', 'request_id': 'req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:51.357285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b70d99ff0094ac6b018c6f8a907e31b', 'request_id': 'req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:51.366198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b70d99ff0094ac6b018c6f8a907e31b', 'request_id': 'req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:51.381027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] The token cache is being invalidate because project 0b70d99ff0094ac6b018c6f8a907e31b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:46:51.381361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:51.382111 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:46:51.398938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1230/2421] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:46:51 2022] DELETE /identity/v3/projects/0b70d99ff0094ac6b018c6f8a907e31b => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:46:54.509233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3281326b-fcd7-45a9-b8e5-8a2a3a08c894 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:54.510111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3281326b-fcd7-45a9-b8e5-8a2a3a08c894 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:54.510111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3281326b-fcd7-45a9-b8e5-8a2a3a08c894 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:54.537321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3281326b-fcd7-45a9-b8e5-8a2a3a08c894 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:46:54.539912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3281326b-fcd7-45a9-b8e5-8a2a3a08c894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:54.636402 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1192/2422] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:46:54 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:46:54.661108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:46:54.667455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:46:54.775804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:46:54.790943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba2c91e-b90f-4dcc-98fa-9accb3585a6c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:46:54.798867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:46:54.800201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:46:54.800551 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:46:54.800902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:46:54.814964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:46:54.817461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:46:54.828641 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1231/2423] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:46:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:47:04.627208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=70,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:04.823765 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:14.868041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:14.868041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:14.868041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:14.900173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:14.901792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:14.911096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1193/2424] 10.222.0.72 () {58 vars in 967 bytes} [Mon Jul 11 14:47:14 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:14.927974 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:14.945871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17750cf-f4e5-443c-8cef-7b6c78172cce service swift] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:14.955005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:14.957070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:14.957910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:14.958385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:14.975901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:14.978067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:14.991457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1232/2425] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:47:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:47:16.060303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:16.063021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:16.139214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca7f007-a05c-4135-af76-5e62b701f33f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:16.143272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47df4009-8e1a-464e-9b86-e5acb995eaf8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:16.143946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47df4009-8e1a-464e-9b86-e5acb995eaf8 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:16.144226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47df4009-8e1a-464e-9b86-e5acb995eaf8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:16.144499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47df4009-8e1a-464e-9b86-e5acb995eaf8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:16.146482 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-47df4009-8e1a-464e-9b86-e5acb995eaf8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:16.220457 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47df4009-8e1a-464e-9b86-e5acb995eaf8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:16.222688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47df4009-8e1a-464e-9b86-e5acb995eaf8 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:16.236596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1194/2426] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:47:19.292015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:19.294272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:19.313264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffff4e5-0abc-4522-8b23-dd760fd134e7 service swift] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:19.320824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:19.322193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:19.322902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:19.323497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] PATH_INFO: `/v3/users/888e7afcc0834d55a8cee3fd6a4b8f44` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:19.325735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=888e7afcc0834d55a8cee3fd6a4b8f44)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:19.329154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:19.413180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '888e7afcc0834d55a8cee3fd6a4b8f44'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:19.431181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '888e7afcc0834d55a8cee3fd6a4b8f44'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:19.432411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '888e7afcc0834d55a8cee3fd6a4b8f44'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:19.448193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '888e7afcc0834d55a8cee3fd6a4b8f44'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:19.472393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '888e7afcc0834d55a8cee3fd6a4b8f44'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:19.483057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1233/2427] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:19 2022] DELETE /identity/v3/users/888e7afcc0834d55a8cee3fd6a4b8f44 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:19.489034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:19.489608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:19.489608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:19.513033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:19.514960 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:19.619888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1195/2428] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:19.636643 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:19.638255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:19.734735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904f49ee-f11a-49b4-bd6a-999c1f3761cc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:19.740174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e179d4-df91-4050-a2f4-1b92f69d57a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:19.740857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e179d4-df91-4050-a2f4-1b92f69d57a7 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:19.741148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e179d4-df91-4050-a2f4-1b92f69d57a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:19.741431 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e179d4-df91-4050-a2f4-1b92f69d57a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:19.755305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e179d4-df91-4050-a2f4-1b92f69d57a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:19.756972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e179d4-df91-4050-a2f4-1b92f69d57a7 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:19.776372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1234/2429] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:20.948142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:20.949784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:20.968258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a146b9-e36d-4fc8-9a74-a44ad2b0293c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:20.976409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:20.977685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:20.978215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:20.978797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] PATH_INFO: `/v3/projects/0f3ad328ff6744e4bcef8d39a4324044` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:20.982787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f3ad328ff6744e4bcef8d39a4324044)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:20.984988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:21.083115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:21.096450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f3ad328ff6744e4bcef8d39a4324044', 'request_id': 'req-df23922d-9c1a-415c-8322-950640f0ac16'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:21.105894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f3ad328ff6744e4bcef8d39a4324044', 'request_id': 'req-df23922d-9c1a-415c-8322-950640f0ac16'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:21.118589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f3ad328ff6744e4bcef8d39a4324044', 'request_id': 'req-df23922d-9c1a-415c-8322-950640f0ac16'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:21.144145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] The token cache is being invalidate because project 0f3ad328ff6744e4bcef8d39a4324044 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:21.145189 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:21.147227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-df23922d-9c1a-415c-8322-950640f0ac16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:21.151400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1196/2430] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:20 2022] DELETE /identity/v3/projects/0f3ad328ff6744e4bcef8d39a4324044 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:21.156428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6fa265-3c44-4128-91af-6f3d54568b88 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:21.157299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6fa265-3c44-4128-91af-6f3d54568b88 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:21.157299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6fa265-3c44-4128-91af-6f3d54568b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:21.195533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ff6fa265-3c44-4128-91af-6f3d54568b88 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:21.197232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ff6fa265-3c44-4128-91af-6f3d54568b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:21.362199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1235/2431] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:21.375239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb27d05-8e2b-43e6-b7e0-b4e5cde7e8fe None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:21.376871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb27d05-8e2b-43e6-b7e0-b4e5cde7e8fe None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:21.377664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb27d05-8e2b-43e6-b7e0-b4e5cde7e8fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:21.415069 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-afb27d05-8e2b-43e6-b7e0-b4e5cde7e8fe None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:21.417915 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-afb27d05-8e2b-43e6-b7e0-b4e5cde7e8fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:21.439834 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1197/2432] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:21.441399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6fa265-3c44-4128-91af-6f3d54568b88 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:21.457006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:21.479294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6fa265-3c44-4128-91af-6f3d54568b88 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:21.500929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:21.500929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:21.501384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:21.501552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:21.502250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:21.502947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:21.505096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:21.521067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1236/2433] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:21.528114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb27d05-8e2b-43e6-b7e0-b4e5cde7e8fe None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:21.546892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb27d05-8e2b-43e6-b7e0-b4e5cde7e8fe None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:21.566386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:21.568057 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:21.568570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:21.569332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:21.570738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:21.591163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:21.688175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1198/2434] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:21 2022] POST /identity/v3/projects => generated 375 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:21.688913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:21.718836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bb4425-9b30-4bf9-8d8c-0af8f3e6d093 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:21.956520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:21.957777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:21.958220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:21.958703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:21.959612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:21.962006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:22.011334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1237/2435] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:21 2022] POST /identity/v3/users => generated 288 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:22.013671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:22.019535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:22.020184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:22.042513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7d348b-6213-4e04-b536-8faf21deb774 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:22.045810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6594c3-21fe-41cd-b2b0-ef83ca98a93e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:22.058139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:22.059064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:22.059916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:22.060237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:22.062041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:22.062301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:22.062663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:22.063136 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] PATH_INFO: `/v3/users/467992b53c4749858fd8b0ccd2db2393` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:22.066220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:22.068584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] RBAC: Authorizing `identity:delete_user(user_id=467992b53c4749858fd8b0ccd2db2393)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:22.070175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:22.070618 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:22.107774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1238/2436] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:22 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:47:22.112741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:22.137328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1811ec7b-76c9-4d21-bc8c-9c733082e8ec admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:22.150575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:22.151406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:22.151698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:22.152072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:22.152967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:22.154775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:22.181849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1239/2437] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:22 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:22.185833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:22.200181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31198810-f54f-4ca6-a9d7-4bd7fd4407bf admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:22.211296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a09cfe-b9f9-4416-9ba3-f81e8ef72252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:22.212514 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a09cfe-b9f9-4416-9ba3-f81e8ef72252 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:22.212946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a09cfe-b9f9-4416-9ba3-f81e8ef72252 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:22.213390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a09cfe-b9f9-4416-9ba3-f81e8ef72252 admin admin] PATH_INFO: `/v3/projects/886e243aa71042c0b11a4444878c600a/users/ab9b328a88294277a7fe957c18488c6f/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:22.225907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '467992b53c4749858fd8b0ccd2db2393'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:22.242865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '467992b53c4749858fd8b0ccd2db2393'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:22.245404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a09cfe-b9f9-4416-9ba3-f81e8ef72252 admin admin] RBAC: Authorizing `identity:create_grant(project_id=886e243aa71042c0b11a4444878c600a, user_id=ab9b328a88294277a7fe957c18488c6f, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:22.248688 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a09cfe-b9f9-4416-9ba3-f81e8ef72252 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:22.260242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '467992b53c4749858fd8b0ccd2db2393'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:22.261392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '467992b53c4749858fd8b0ccd2db2393'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:22.285515 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '467992b53c4749858fd8b0ccd2db2393'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:22.286911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1240/2438] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:22 2022] PUT /identity/v3/projects/886e243aa71042c0b11a4444878c600a/users/ab9b328a88294277a7fe957c18488c6f/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:22.298049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1199/2439] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:22 2022] DELETE /identity/v3/users/467992b53c4749858fd8b0ccd2db2393 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:22.301103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:22.302194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:22.303429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:22.382868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] MFA Rules not processed for user `ab9b328a88294277a7fe957c18488c6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:22.384770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:22.589367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1241/2440] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:22 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:22.606086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:22.608032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:22.644737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:22.648227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:22.724054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:22.775475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea23ab-3480-4afc-bf6d-5a15b2db691d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:22.786983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17a2a68-fd66-4fce-bd42-052cd702c678 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:22.788155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17a2a68-fd66-4fce-bd42-052cd702c678 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:22.788887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17a2a68-fd66-4fce-bd42-052cd702c678 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:22.789400 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17a2a68-fd66-4fce-bd42-052cd702c678 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:22.810586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17a2a68-fd66-4fce-bd42-052cd702c678 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:22.813776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17a2a68-fd66-4fce-bd42-052cd702c678 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:22.846516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3879bcf-62a0-43cc-951c-ed3d6ccc09b0 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:22.848178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1200/2441] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:22 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:22.858252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:22.860359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:22.860868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:22.861611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] PATH_INFO: `/v3/projects/c0e363b1584848c1aecbe186cf91eec5` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:22.865293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0e363b1584848c1aecbe186cf91eec5)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:22.872966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:23.003520 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0e363b1584848c1aecbe186cf91eec5', 'request_id': 'req-08af2e0f-45be-4566-8747-a9692b405400'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:23.017423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0e363b1584848c1aecbe186cf91eec5', 'request_id': 'req-08af2e0f-45be-4566-8747-a9692b405400'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:23.028569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0e363b1584848c1aecbe186cf91eec5', 'request_id': 'req-08af2e0f-45be-4566-8747-a9692b405400'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:23.063928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] The token cache is being invalidate because project c0e363b1584848c1aecbe186cf91eec5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:23.065571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:23.067485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-08af2e0f-45be-4566-8747-a9692b405400 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:23.077733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1242/2442] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:22 2022] DELETE /identity/v3/projects/c0e363b1584848c1aecbe186cf91eec5 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:23.085613 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec594464-fc33-41ed-bf99-4fd74046cb6a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.087723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec594464-fc33-41ed-bf99-4fd74046cb6a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.088463 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec594464-fc33-41ed-bf99-4fd74046cb6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.115676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-ec594464-fc33-41ed-bf99-4fd74046cb6a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:23.117185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ec594464-fc33-41ed-bf99-4fd74046cb6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:23.131588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:23.215851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1201/2443] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:23.229947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e5c4f-97fb-45e4-8b5d-06d199c72d54 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.230564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e5c4f-97fb-45e4-8b5d-06d199c72d54 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.230921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e5c4f-97fb-45e4-8b5d-06d199c72d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.257320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-775e5c4f-97fb-45e4-8b5d-06d199c72d54 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:23.259714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-775e5c4f-97fb-45e4-8b5d-06d199c72d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:23.273325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1243/2444] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:23.278533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec594464-fc33-41ed-bf99-4fd74046cb6a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:23.297695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec594464-fc33-41ed-bf99-4fd74046cb6a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:23.309501 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:23.311614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.312083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.312502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.316005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:23.316653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:23.321196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:23.351989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1202/2445] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:23.358345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e5c4f-97fb-45e4-8b5d-06d199c72d54 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:23.376605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e5c4f-97fb-45e4-8b5d-06d199c72d54 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:23.386733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:23.387885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.388407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.388914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.390245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:23.394284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:23.437564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1244/2446] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:23 2022] POST /identity/v3/projects => generated 369 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:23.445725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:23.468125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae84a4e7-79d5-4aa0-943e-14c1c5cf0dc1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:23.583743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:23.584888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.585393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.585890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.587303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:23.591285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:23.626237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1203/2447] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:23 2022] POST /identity/v3/users => generated 285 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:23.631639 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:23.649446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc20957b-c617-494d-8521-b7d174fc83fd admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:23.656506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:23.657722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.658982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.659487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.661105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:23.663772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:23.683062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1245/2448] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:23 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:23.688331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:23.704008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0bc363-008f-49b8-b655-16cc4e48836e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:23.709372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56abae38-4fb2-48be-ac55-32845dac6a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:23.710270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abae38-4fb2-48be-ac55-32845dac6a9d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.710599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abae38-4fb2-48be-ac55-32845dac6a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.710912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abae38-4fb2-48be-ac55-32845dac6a9d admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.711666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56abae38-4fb2-48be-ac55-32845dac6a9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:23.713329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56abae38-4fb2-48be-ac55-32845dac6a9d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:23.725986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1204/2449] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:23 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:23.731722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:23.743119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab251574-2754-49eb-a87e-1b230c49a637 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:23.748567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:23.749501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.749782 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.750057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] PATH_INFO: `/v3/projects/c753b036247d4e06a226ea46d484d83c/users/27bd1e86ebeb479888ecd7c6784c9f18/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.769934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c753b036247d4e06a226ea46d484d83c, user_id=27bd1e86ebeb479888ecd7c6784c9f18, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:23.772746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:23.788527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1246/2450] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:23 2022] PUT /identity/v3/projects/c753b036247d4e06a226ea46d484d83c/users/27bd1e86ebeb479888ecd7c6784c9f18/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:23.799207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f39fe-e592-4c62-ade5-cdf7b4aa799a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:23.799522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f39fe-e592-4c62-ade5-cdf7b4aa799a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:23.799688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f39fe-e592-4c62-ade5-cdf7b4aa799a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:23.833034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-211f39fe-e592-4c62-ade5-cdf7b4aa799a None None] MFA Rules not processed for user `27bd1e86ebeb479888ecd7c6784c9f18`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:23.834517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-211f39fe-e592-4c62-ade5-cdf7b4aa799a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:23.912508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1205/2451] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:23 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:23.917572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:23.974847 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:23.994827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc9efff-5207-440e-bb7c-2f5f184c5f69 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:24.000992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:24.001763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.002099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.002421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.003233 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:24.005592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:24.036043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1247/2452] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:23 2022] POST /identity/v3/projects => generated 367 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:24.040769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f39fe-e592-4c62-ade5-cdf7b4aa799a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:24.053620 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f39fe-e592-4c62-ade5-cdf7b4aa799a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:24.059130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:24.122192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:24.123108 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.123406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.123727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.124470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:24.127143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:24.149850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1206/2453] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:24 2022] POST /identity/v3/users => generated 290 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:24.155840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:24.167934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac687d3-39d8-4aae-b64b-8dbe006aeb28 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:24.172983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:24.173714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.174003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.174286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.175078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:24.176757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:24.189102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1248/2454] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:24.195909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:24.208291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7962100f-eb05-465a-a46d-5b0855ce0b0e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:24.213532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:24.214486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.214777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.215058 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] PATH_INFO: `/v3/projects/cbe424a0a3274fb29220db9a6d48823d/users/f02a6931d81347078240a9a53942e9db/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.232301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbe424a0a3274fb29220db9a6d48823d, user_id=f02a6931d81347078240a9a53942e9db, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:24.235116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:24.250576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1207/2455] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:24 2022] PUT /identity/v3/projects/cbe424a0a3274fb29220db9a6d48823d/users/f02a6931d81347078240a9a53942e9db/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:24.255741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:24.266683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153ba800-fea6-4441-ab72-59d545d2e147 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:24.333128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae1dd12-3d55-41cc-9a9e-b2d90340ce2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:24.333907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae1dd12-3d55-41cc-9a9e-b2d90340ce2f admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.334195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae1dd12-3d55-41cc-9a9e-b2d90340ce2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.334513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae1dd12-3d55-41cc-9a9e-b2d90340ce2f admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.335270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae1dd12-3d55-41cc-9a9e-b2d90340ce2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:24.337018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae1dd12-3d55-41cc-9a9e-b2d90340ce2f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:24.349539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1249/2456] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:24.356218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:24.369610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f6e1c0-f2fc-4a42-81b0-269540f55898 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:24.378054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:24.379601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.380810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.381265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] PATH_INFO: `/v3/projects/cbe424a0a3274fb29220db9a6d48823d/users/f02a6931d81347078240a9a53942e9db/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.385342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbe424a0a3274fb29220db9a6d48823d, user_id=f02a6931d81347078240a9a53942e9db, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:24.390279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:24.410454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1208/2457] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:24 2022] PUT /identity/v3/projects/cbe424a0a3274fb29220db9a6d48823d/users/f02a6931d81347078240a9a53942e9db/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:24.438642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691c08a-99f6-4c20-ae0b-afda72f259bb None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.438642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691c08a-99f6-4c20-ae0b-afda72f259bb None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.438642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691c08a-99f6-4c20-ae0b-afda72f259bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.489171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5691c08a-99f6-4c20-ae0b-afda72f259bb None None] MFA Rules not processed for user `f02a6931d81347078240a9a53942e9db`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:24.489171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5691c08a-99f6-4c20-ae0b-afda72f259bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:24.625964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1250/2458] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:24 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:24.676369 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:24.684027 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:24.821354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288b44f5-4c0a-4591-b51b-af00bc3c93dd admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:24.828157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be3aaf4-c7d3-4276-b215-6d8cefa9beb4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:24.829253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be3aaf4-c7d3-4276-b215-6d8cefa9beb4 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:24.829758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be3aaf4-c7d3-4276-b215-6d8cefa9beb4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:24.830238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be3aaf4-c7d3-4276-b215-6d8cefa9beb4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:24.850919 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be3aaf4-c7d3-4276-b215-6d8cefa9beb4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:24.852430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be3aaf4-c7d3-4276-b215-6d8cefa9beb4 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:24.864506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1209/2459] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:24 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:26.845526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691c08a-99f6-4c20-ae0b-afda72f259bb None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:26.925248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:26.969303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691c08a-99f6-4c20-ae0b-afda72f259bb None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:26.975544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:26.976366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:26.976857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:26.977145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] PATH_INFO: `/v3/users/27bd1e86ebeb479888ecd7c6784c9f18` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:26.978975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] RBAC: Authorizing `identity:delete_user(user_id=27bd1e86ebeb479888ecd7c6784c9f18)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:26.981012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:27.046916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27bd1e86ebeb479888ecd7c6784c9f18'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:27.061699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27bd1e86ebeb479888ecd7c6784c9f18'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:27.065058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27bd1e86ebeb479888ecd7c6784c9f18'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:27.079790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27bd1e86ebeb479888ecd7c6784c9f18'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:27.099931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27bd1e86ebeb479888ecd7c6784c9f18'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:27.109015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1251/2460] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:26 2022] DELETE /identity/v3/users/27bd1e86ebeb479888ecd7c6784c9f18 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:27.114692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:27.115531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:27.115531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:27.140694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:27.142299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:27.205759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:27.208258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:27.253989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1210/2461] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:27.269643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:27.271320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:27.279699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:27.313367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d986fc-fa7a-4021-8524-c05e9f54266e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:27.320045 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:27.321173 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:27.321659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:27.322137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:27.325632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:27.347961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:27.350640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d36ada3-ffcc-436d-8fc3-37e84cb446f6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:27.350860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:27.356851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:27.357896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:27.358288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:27.358777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:27.369756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1252/2462] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:47:27.375699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:27.377392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:27.390686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1211/2463] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:47:27.846534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:27.848961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:27.871703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf094b6-1a36-4328-9490-ed11860c4550 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:27.883976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:27.885444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:27.885987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:27.886544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] PATH_INFO: `/v3/projects/c753b036247d4e06a226ea46d484d83c` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:27.888905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c753b036247d4e06a226ea46d484d83c)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:27.892697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:27.973038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c753b036247d4e06a226ea46d484d83c', 'request_id': 'req-338953c2-ea94-46bb-945b-b29f1ba7d50f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:27.988304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c753b036247d4e06a226ea46d484d83c', 'request_id': 'req-338953c2-ea94-46bb-945b-b29f1ba7d50f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.002000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:28.005083 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c753b036247d4e06a226ea46d484d83c', 'request_id': 'req-338953c2-ea94-46bb-945b-b29f1ba7d50f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.036019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] The token cache is being invalidate because project c753b036247d4e06a226ea46d484d83c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:28.036801 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.038121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.041822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1253/2464] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:27 2022] DELETE /identity/v3/projects/c753b036247d4e06a226ea46d484d83c => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:28.046093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:28.068303 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b0abb8-7511-4bc0-a958-98017a992092 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:28.190167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:28.191807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:28.192295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:28.192783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] PATH_INFO: `/v3/users/f02a6931d81347078240a9a53942e9db` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:28.195093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f02a6931d81347078240a9a53942e9db)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:28.199573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:28.304115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f02a6931d81347078240a9a53942e9db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.346628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f02a6931d81347078240a9a53942e9db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.359328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f02a6931d81347078240a9a53942e9db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.360410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f02a6931d81347078240a9a53942e9db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.389602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a2141a5e-fa1b-4d08-93e4-98cc54c72408 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f02a6931d81347078240a9a53942e9db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:28.399232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1212/2465] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:28 2022] DELETE /identity/v3/users/f02a6931d81347078240a9a53942e9db => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:28.946064 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:28.948660 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:28.979673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338953c2-ea94-46bb-945b-b29f1ba7d50f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:29.081040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:29.084486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:29.086133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:29.086845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] PATH_INFO: `/v3/projects/cbe424a0a3274fb29220db9a6d48823d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:29.091640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] RBAC: Authorizing `identity:delete_project(project_id=cbe424a0a3274fb29220db9a6d48823d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:29.098344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:29.231367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cbe424a0a3274fb29220db9a6d48823d', 'request_id': 'req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:29.243542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cbe424a0a3274fb29220db9a6d48823d', 'request_id': 'req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:29.261966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cbe424a0a3274fb29220db9a6d48823d', 'request_id': 'req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:29.288384 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] The token cache is being invalidate because project cbe424a0a3274fb29220db9a6d48823d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:29.288921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:29.290084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-efc5e7a7-62a6-4b2c-8104-db25b9511eaa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:29.303917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280afd06-79b3-42aa-b9ae-99a9848d346f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:29.304450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280afd06-79b3-42aa-b9ae-99a9848d346f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:29.304565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280afd06-79b3-42aa-b9ae-99a9848d346f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:29.305987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1254/2466] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:28 2022] DELETE /identity/v3/projects/cbe424a0a3274fb29220db9a6d48823d => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:29.359975 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-280afd06-79b3-42aa-b9ae-99a9848d346f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:29.362762 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-280afd06-79b3-42aa-b9ae-99a9848d346f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:29.420709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:29.499029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1213/2467] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:29.508181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b581c1-3557-4639-95b1-21e639a5c8af None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:29.508472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b581c1-3557-4639-95b1-21e639a5c8af None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:29.508867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b581c1-3557-4639-95b1-21e639a5c8af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:29.535135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-37b581c1-3557-4639-95b1-21e639a5c8af None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:29.536480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-37b581c1-3557-4639-95b1-21e639a5c8af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:29.553012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280afd06-79b3-42aa-b9ae-99a9848d346f None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:29.554058 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1255/2468] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:29.564648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280afd06-79b3-42aa-b9ae-99a9848d346f None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:29.569820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:29.570682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:29.570979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:29.571251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:29.572404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:29.572726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:29.574517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:29.588815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1214/2469] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:29.594084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b581c1-3557-4639-95b1-21e639a5c8af None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:29.613843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b581c1-3557-4639-95b1-21e639a5c8af None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:29.622481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:29.623371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:29.624259 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:29.624418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:29.626913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:29.629790 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:29.698368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1256/2470] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:29 2022] POST /identity/v3/projects => generated 377 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:29.703568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:29.715821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4908cc02-fe41-4ad1-82c0-a488f3958804 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:29.855854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:29.857311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:29.857838 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:29.858348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:29.860182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:29.864547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:29.913227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1215/2471] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:29 2022] POST /identity/v3/users => generated 289 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:29.919928 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:29.943258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac540dc9-fd35-4b75-8c4c-592d0007fc79 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:29.956295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:29.958132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:29.959500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:29.959500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:29.960871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:29.964289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:29.996153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1257/2472] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:29 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:30.003201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:30.026636 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ed6a2-f77d-4ed9-8978-50e49fe0631f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:30.035444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56521389-03e4-4c81-8bb0-f1f839828ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:30.036580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56521389-03e4-4c81-8bb0-f1f839828ca9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:30.037097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56521389-03e4-4c81-8bb0-f1f839828ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:30.037594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56521389-03e4-4c81-8bb0-f1f839828ca9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:30.039475 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56521389-03e4-4c81-8bb0-f1f839828ca9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:30.042655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56521389-03e4-4c81-8bb0-f1f839828ca9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:30.064877 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1216/2473] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:30.072224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:30.104056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c2c1a-e954-49e4-9fe8-0203c7dc385c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:30.113667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:30.114892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:30.115394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:30.115894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] PATH_INFO: `/v3/projects/d457ae6b47fb4ab28854c930b3210924/users/35c5e0aac19d4a6dbdb872e738b222db/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:30.165039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d457ae6b47fb4ab28854c930b3210924, user_id=35c5e0aac19d4a6dbdb872e738b222db, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:30.174174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:30.200712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1258/2474] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:30 2022] PUT /identity/v3/projects/d457ae6b47fb4ab28854c930b3210924/users/35c5e0aac19d4a6dbdb872e738b222db/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:30.218177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:30.218662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:30.219137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:30.268733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] MFA Rules not processed for user `35c5e0aac19d4a6dbdb872e738b222db`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:30.270474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:30.405358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1217/2475] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:30 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:30.411613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:30.472581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:30.605158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28751498-6003-4fc8-83c3-3c9f5c6ca553 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:30.619151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:30.623618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:30.623618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:30.623618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:30.623618 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:30.629191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:30.669996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:30.678305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:30.709521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1259/2476] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:30 2022] POST /identity/v3/projects => generated 377 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:30.710581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:30.737705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:30.824829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4baf0b47-955f-4ec4-8206-4edd1d916938 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:30.877182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b601aead-662e-48e1-b9af-04c968a4ff10 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:30.885160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:30.886127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:30.886669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:30.887142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:30.888582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:30.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:30.931169 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:30.932285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:30.932285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:30.932859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:30.936255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:30.937702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1260/2477] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:30 2022] POST /identity/v3/users => generated 295 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:30.951244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:30.971902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8485f9-a0e5-4636-a862-0490ca6cfec3 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:30.979087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:30.980121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:30.980408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:30.981501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:30.982525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:30.984412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:31.014134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1261/2478] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:31.018172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:31.052966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5ea077-f210-457a-84aa-3f0e794b46b8 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:31.070794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:31.074085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:31.079175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:31.079175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] PATH_INFO: `/v3/projects/3cda26fef7f74e2ab8e622f278bae40d/users/e10d5e9102d04dcd9c46792c6f20fa4a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:31.138156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cda26fef7f74e2ab8e622f278bae40d, user_id=e10d5e9102d04dcd9c46792c6f20fa4a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:31.157875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:31.189947 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1262/2479] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:31 2022] PUT /identity/v3/projects/3cda26fef7f74e2ab8e622f278bae40d/users/e10d5e9102d04dcd9c46792c6f20fa4a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:31.194926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:31.208610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:31.211925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:31.268363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1218/2480] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:30 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 599 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:31.377568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:31.393013 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887d0b6f-1fa7-457f-8cfd-fbb0b7326570 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:31.408615 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d780b13-2cc2-4bb2-a5bb-dd9c76faf440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:31.409960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d780b13-2cc2-4bb2-a5bb-dd9c76faf440 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:31.410495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d780b13-2cc2-4bb2-a5bb-dd9c76faf440 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:31.411003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d780b13-2cc2-4bb2-a5bb-dd9c76faf440 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:31.413170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d780b13-2cc2-4bb2-a5bb-dd9c76faf440 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:31.415793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d780b13-2cc2-4bb2-a5bb-dd9c76faf440 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:31.448391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1263/2481] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:31 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:31.465332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:31.491467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7698464e-3512-4c6d-9c34-de36ccab9293 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:31.516221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:31.517319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:31.517680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:31.518014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] PATH_INFO: `/v3/projects/3cda26fef7f74e2ab8e622f278bae40d/users/e10d5e9102d04dcd9c46792c6f20fa4a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:31.526095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cda26fef7f74e2ab8e622f278bae40d, user_id=e10d5e9102d04dcd9c46792c6f20fa4a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:31.528987 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:31.571034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1219/2482] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:31 2022] PUT /identity/v3/projects/3cda26fef7f74e2ab8e622f278bae40d/users/e10d5e9102d04dcd9c46792c6f20fa4a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:31.583537 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:31.584288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:31.584949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:31.672315 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] MFA Rules not processed for user `e10d5e9102d04dcd9c46792c6f20fa4a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:31.674223 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:31.942483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1264/2483] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:31 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 363 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:31.956263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:31.960335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:32.119599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a31f7c-573d-4592-ab0e-f23c7c38eda9 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:32.127545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:32.128843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:32.129421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:32.130287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:32.225164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:32.263209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:32.265392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:32.277254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1220/2484] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:31 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:32.793353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:32.795094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:32.882624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a0b25-a6a1-4776-8312-e33a8a0326a8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:32.887873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:32.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:32.889426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:32.889426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] PATH_INFO: `/v3/users/3ae56fd1172f4bf8806eefc6e8877fef` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:32.891577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ae56fd1172f4bf8806eefc6e8877fef)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:32.899526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:32.957255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ae56fd1172f4bf8806eefc6e8877fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:32.973372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ae56fd1172f4bf8806eefc6e8877fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:32.974288 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ae56fd1172f4bf8806eefc6e8877fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:32.983661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ae56fd1172f4bf8806eefc6e8877fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:32.997390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:32.999401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-78a73d81-b7f5-4a9d-a634-36ef4feaa161 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ae56fd1172f4bf8806eefc6e8877fef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:33.009367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1265/2485] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:32 2022] DELETE /identity/v3/users/3ae56fd1172f4bf8806eefc6e8877fef => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:33.221038 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:33.239721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0f831b-ebac-4cc0-b125-37a2db6cf803 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:33.312161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:33.313076 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.313370 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.313642 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] PATH_INFO: `/v3/projects/1baa6ea2e19747d1af1e1042c362895d` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.315533 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1baa6ea2e19747d1af1e1042c362895d)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:33.317489 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:33.373247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1baa6ea2e19747d1af1e1042c362895d', 'request_id': 'req-59653adc-4946-4c8f-bb32-8414822c4744'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:33.381435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1baa6ea2e19747d1af1e1042c362895d', 'request_id': 'req-59653adc-4946-4c8f-bb32-8414822c4744'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:33.390791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1baa6ea2e19747d1af1e1042c362895d', 'request_id': 'req-59653adc-4946-4c8f-bb32-8414822c4744'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:33.404902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] The token cache is being invalidate because project 1baa6ea2e19747d1af1e1042c362895d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:33.405205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:33.405888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-59653adc-4946-4c8f-bb32-8414822c4744 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:33.407550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1221/2486] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:33 2022] DELETE /identity/v3/projects/1baa6ea2e19747d1af1e1042c362895d => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:33.413560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b0b70-a6ff-45f2-b558-2b1f3b390eb1 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.413990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b0b70-a6ff-45f2-b558-2b1f3b390eb1 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.414337 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021b0b70-a6ff-45f2-b558-2b1f3b390eb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.436429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-021b0b70-a6ff-45f2-b558-2b1f3b390eb1 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:33.437691 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-021b0b70-a6ff-45f2-b558-2b1f3b390eb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:33.510310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1266/2487] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:33.519116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e877e81-cfeb-491b-9a62-6f7d35451623 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.519406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e877e81-cfeb-491b-9a62-6f7d35451623 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.519673 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e877e81-cfeb-491b-9a62-6f7d35451623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.540426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2e877e81-cfeb-491b-9a62-6f7d35451623 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:33.541744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2e877e81-cfeb-491b-9a62-6f7d35451623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:33.552706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1222/2488] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:33.557363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021b0b70-a6ff-45f2-b558-2b1f3b390eb1 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:33.568824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021b0b70-a6ff-45f2-b558-2b1f3b390eb1 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:33.573977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:33.574759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.575094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.575465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.576309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:33.576657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:33.578381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:33.592225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1267/2489] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:33.596972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e877e81-cfeb-491b-9a62-6f7d35451623 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:33.607509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e877e81-cfeb-491b-9a62-6f7d35451623 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:33.613624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:33.614394 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.614778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.615297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.616097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:33.618332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:33.657697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1223/2490] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:33 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:33.661981 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:33.677155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f002ff5-993d-4c38-a29d-13fafa48ae2a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:33.747703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:33.748402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.748689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.748969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.749699 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:33.752309 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:33.777070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1268/2491] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:33 2022] POST /identity/v3/users => generated 287 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:33.782078 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:33.793857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74051294-eb76-4318-a043-5b2b660d1490 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:33.800733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:33.801436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.801723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.802002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.802793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:33.804460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:33.817277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1224/2492] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:33.823721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:33.838953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3388a02b-33db-4feb-ae23-b68800b546a0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:33.844294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f721025-658f-458b-ae79-1cd82f313fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:33.845150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f721025-658f-458b-ae79-1cd82f313fa3 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.845600 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f721025-658f-458b-ae79-1cd82f313fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.846031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f721025-658f-458b-ae79-1cd82f313fa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.846951 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f721025-658f-458b-ae79-1cd82f313fa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:33.849172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f721025-658f-458b-ae79-1cd82f313fa3 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:33.864378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1269/2493] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:33 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:33.869427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:33.883322 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5573df-bf71-4aa7-a9dd-3e58847a0a7f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:33.889137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:33.890195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.890629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.891086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] PATH_INFO: `/v3/projects/ab6df275ee124e1a90efacdf16c6fa2e/users/6352e8010a3d4163a5b9c3fd6d68b37b/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.912649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab6df275ee124e1a90efacdf16c6fa2e, user_id=6352e8010a3d4163a5b9c3fd6d68b37b, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:33.916087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:33.928996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:33.935495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1225/2494] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:33 2022] PUT /identity/v3/projects/ab6df275ee124e1a90efacdf16c6fa2e/users/6352e8010a3d4163a5b9c3fd6d68b37b/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:33.946893 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448603b3-91f5-4ae5-9a56-2fff8debdaab None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:33.947183 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448603b3-91f5-4ae5-9a56-2fff8debdaab None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:33.947463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448603b3-91f5-4ae5-9a56-2fff8debdaab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:33.978768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-448603b3-91f5-4ae5-9a56-2fff8debdaab None None] MFA Rules not processed for user `6352e8010a3d4163a5b9c3fd6d68b37b`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:33.980081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-448603b3-91f5-4ae5-9a56-2fff8debdaab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:34.037427 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:34.059656 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1270/2495] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:34.071998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:34.074011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:34.161110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cce77a-5dc7-43b0-84b3-047133b25ed1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:34.167632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb34e637-e49b-4479-a155-ab1c54161049 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:34.168428 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34e637-e49b-4479-a155-ab1c54161049 service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:34.168860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34e637-e49b-4479-a155-ab1c54161049 service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:34.169236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34e637-e49b-4479-a155-ab1c54161049 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:34.179229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb34e637-e49b-4479-a155-ab1c54161049 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:34.180882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb34e637-e49b-4479-a155-ab1c54161049 service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:34.191360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1226/2496] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:34 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:34.463332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448603b3-91f5-4ae5-9a56-2fff8debdaab None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:34.536162 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448603b3-91f5-4ae5-9a56-2fff8debdaab None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:34.542015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:34.542845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:34.543129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:34.543407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] PATH_INFO: `/v3/users/6352e8010a3d4163a5b9c3fd6d68b37b` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:34.544746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6352e8010a3d4163a5b9c3fd6d68b37b)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:34.546777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:34.615166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6352e8010a3d4163a5b9c3fd6d68b37b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:34.625503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6352e8010a3d4163a5b9c3fd6d68b37b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:34.626153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6352e8010a3d4163a5b9c3fd6d68b37b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:34.636825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6352e8010a3d4163a5b9c3fd6d68b37b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:34.652310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6352e8010a3d4163a5b9c3fd6d68b37b'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:34.662697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1271/2497] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:34 2022] DELETE /identity/v3/users/6352e8010a3d4163a5b9c3fd6d68b37b => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:34.667572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:34.668161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:34.668407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:34.689848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:34.691129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:34.772789 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1227/2498] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:34.783658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:34.785395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:34.857766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cc30d1-604f-4a59-916c-562aaefe564a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:34.863807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492574f4-2b2b-4d48-a134-036f97a1830a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:34.865169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492574f4-2b2b-4d48-a134-036f97a1830a service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:34.865658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492574f4-2b2b-4d48-a134-036f97a1830a service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:34.866220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492574f4-2b2b-4d48-a134-036f97a1830a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:34.887262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492574f4-2b2b-4d48-a134-036f97a1830a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:34.888849 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492574f4-2b2b-4d48-a134-036f97a1830a service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:34.917770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1272/2499] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:35.405444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:35.408614 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:35.434139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbac9d97-f2da-4616-9c47-5217a9770a16 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:35.447458 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:35.448604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:35.448961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:35.449564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] PATH_INFO: `/v3/projects/ab6df275ee124e1a90efacdf16c6fa2e` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:35.452376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab6df275ee124e1a90efacdf16c6fa2e)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:35.456407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:35.529616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab6df275ee124e1a90efacdf16c6fa2e', 'request_id': 'req-f17cf538-1c05-4961-922d-c8fbdd211ad7'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:35.550168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab6df275ee124e1a90efacdf16c6fa2e', 'request_id': 'req-f17cf538-1c05-4961-922d-c8fbdd211ad7'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:35.567578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab6df275ee124e1a90efacdf16c6fa2e', 'request_id': 'req-f17cf538-1c05-4961-922d-c8fbdd211ad7'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:35.596213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] The token cache is being invalidate because project ab6df275ee124e1a90efacdf16c6fa2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:35.596781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:35.606213 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f17cf538-1c05-4961-922d-c8fbdd211ad7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:35.610393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1228/2500] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:35 2022] DELETE /identity/v3/projects/ab6df275ee124e1a90efacdf16c6fa2e => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:35.619549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d32e09-3593-483d-9979-02155624b5dd None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:35.620449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d32e09-3593-483d-9979-02155624b5dd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:35.621476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d32e09-3593-483d-9979-02155624b5dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:35.652189 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-41d32e09-3593-483d-9979-02155624b5dd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:35.653371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-41d32e09-3593-483d-9979-02155624b5dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:35.668856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:35.734807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1273/2501] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:35.743349 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5374408-5066-4439-9813-ae15385dfda6 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:35.743930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5374408-5066-4439-9813-ae15385dfda6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:35.743930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5374408-5066-4439-9813-ae15385dfda6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:35.763737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e5374408-5066-4439-9813-ae15385dfda6 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:35.765005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e5374408-5066-4439-9813-ae15385dfda6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:35.774782 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1229/2502] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:35.779879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d32e09-3593-483d-9979-02155624b5dd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:35.794217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d32e09-3593-483d-9979-02155624b5dd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:35.799390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:35.800075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:35.800357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:35.800655 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:35.801545 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:35.801860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:35.803575 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:35.820888 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1274/2503] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:35.822180 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5374408-5066-4439-9813-ae15385dfda6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:35.827073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:35.836553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5374408-5066-4439-9813-ae15385dfda6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:35.844116 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:35.845152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:35.846097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:35.846600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:35.847438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:35.849545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:35.887780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1230/2504] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:35 2022] POST /identity/v3/projects => generated 403 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:35.892738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:35.903584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513ead0-5b11-48e1-97cb-d4fe586a5e83 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:35.970222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:35.971480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:35.971803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:35.972082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:35.972799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:35.974783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:35.996554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1275/2505] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:35 2022] POST /identity/v3/users => generated 302 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:36.001095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.012619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cf786c-16ef-4c6b-82df-34f7fa5eecad admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.018600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.019304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.019595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.019898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.020644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.022401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.035112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1231/2506] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:35 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:36.039529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.050175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11018f12-46de-443f-8246-a1ba468d805e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.055126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beeac1a-3e2a-4cc5-856b-18564fc1b4e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.055822 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beeac1a-3e2a-4cc5-856b-18564fc1b4e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.056110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beeac1a-3e2a-4cc5-856b-18564fc1b4e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.056391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beeac1a-3e2a-4cc5-856b-18564fc1b4e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.057150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beeac1a-3e2a-4cc5-856b-18564fc1b4e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.058845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beeac1a-3e2a-4cc5-856b-18564fc1b4e2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.070028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1276/2507] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:36.074888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.086947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1331d361-1295-434e-b583-78b94ff22f49 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.092413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.093368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.093649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.093927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] PATH_INFO: `/v3/projects/04e89378a1734f6c881e0e6234986a6b/users/729ff55f74c94f9794e5d1da8471def8/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.111047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=04e89378a1734f6c881e0e6234986a6b, user_id=729ff55f74c94f9794e5d1da8471def8, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.113924 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.130540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1232/2508] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:36 2022] PUT /identity/v3/projects/04e89378a1734f6c881e0e6234986a6b/users/729ff55f74c94f9794e5d1da8471def8/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:36.143124 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e367db-ff6a-4f55-af1d-6fa887f2e52e None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.143421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e367db-ff6a-4f55-af1d-6fa887f2e52e None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.143733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e367db-ff6a-4f55-af1d-6fa887f2e52e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.173868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f8e367db-ff6a-4f55-af1d-6fa887f2e52e None None] MFA Rules not processed for user `729ff55f74c94f9794e5d1da8471def8`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:36.175103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f8e367db-ff6a-4f55-af1d-6fa887f2e52e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:36.249809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1277/2509] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:36 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:36.255821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.330676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcf58fd-912c-4b21-bcd7-c16edb9711fd admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.336089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.336781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.337065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.337342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.338064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.340105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.368291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1233/2510] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:36 2022] POST /identity/v3/projects => generated 405 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:36.373333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e367db-ff6a-4f55-af1d-6fa887f2e52e None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.386353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e367db-ff6a-4f55-af1d-6fa887f2e52e None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.391762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:36.462112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.462873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.463156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.463437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.464163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.466313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.489952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1278/2511] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:36 2022] POST /identity/v3/users => generated 309 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:36.494245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.505566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708ef06e-4295-49f9-9549-451e6bebda6a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.510483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.511184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.511474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.511759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.512507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.514204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.527312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1234/2512] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:36.531800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.543721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5577ad86-10ca-4325-8f82-714d4afec7a5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.548812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.549743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.550033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.550319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] PATH_INFO: `/v3/projects/85f7503b5c774d39b1ce03f364e9104f/users/4b016045e1584f3c9ac7608965ce3b12/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.568026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85f7503b5c774d39b1ce03f364e9104f, user_id=4b016045e1584f3c9ac7608965ce3b12, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.571107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.587931 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1279/2513] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:36 2022] PUT /identity/v3/projects/85f7503b5c774d39b1ce03f364e9104f/users/4b016045e1584f3c9ac7608965ce3b12/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:36.592525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.604348 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b39d07-7ac1-498e-8a12-f13475acc848 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.699957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f670aed-3b0c-4cea-b1cc-72770d447d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.700724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f670aed-3b0c-4cea-b1cc-72770d447d53 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.701143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f670aed-3b0c-4cea-b1cc-72770d447d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.701516 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f670aed-3b0c-4cea-b1cc-72770d447d53 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.702378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f670aed-3b0c-4cea-b1cc-72770d447d53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.704236 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f670aed-3b0c-4cea-b1cc-72770d447d53 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.717201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1235/2514] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:36.724917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.740314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c11619-ac9d-4b9d-aa51-57f20965fd94 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:36.745215 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:36.746129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.746414 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.746794 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] PATH_INFO: `/v3/projects/85f7503b5c774d39b1ce03f364e9104f/users/4b016045e1584f3c9ac7608965ce3b12/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.748912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85f7503b5c774d39b1ce03f364e9104f, user_id=4b016045e1584f3c9ac7608965ce3b12, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:36.751717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:36.764403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1280/2515] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:36 2022] PUT /identity/v3/projects/85f7503b5c774d39b1ce03f364e9104f/users/4b016045e1584f3c9ac7608965ce3b12/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:36.776421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859f38e-4052-46e2-a1a1-e5611ede9333 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:36.777091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859f38e-4052-46e2-a1a1-e5611ede9333 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:36.777091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859f38e-4052-46e2-a1a1-e5611ede9333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:36.809292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a859f38e-4052-46e2-a1a1-e5611ede9333 None None] MFA Rules not processed for user `4b016045e1584f3c9ac7608965ce3b12`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:36.810683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a859f38e-4052-46e2-a1a1-e5611ede9333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:36.815966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:36.901406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1236/2516] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:36 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:36.912580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:36.914865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:37.025963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b17c5-6bc0-4761-8af2-98a25155a745 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:37.030283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e047f12d-fe07-46e1-8af7-ea8b0250bda2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:37.031005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e047f12d-fe07-46e1-8af7-ea8b0250bda2 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:37.031292 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e047f12d-fe07-46e1-8af7-ea8b0250bda2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:37.031570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e047f12d-fe07-46e1-8af7-ea8b0250bda2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:37.042687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e047f12d-fe07-46e1-8af7-ea8b0250bda2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:37.044176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e047f12d-fe07-46e1-8af7-ea8b0250bda2 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:37.056042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1281/2517] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:36 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:38.151045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a859f38e-4052-46e2-a1a1-e5611ede9333 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:38.227131 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a859f38e-4052-46e2-a1a1-e5611ede9333 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:38.231977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:38.233229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:38.233514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:38.234202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] PATH_INFO: `/v3/users/729ff55f74c94f9794e5d1da8471def8` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:38.236283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] RBAC: Authorizing `identity:delete_user(user_id=729ff55f74c94f9794e5d1da8471def8)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:38.238934 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:38.296976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '729ff55f74c94f9794e5d1da8471def8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:38.311646 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '729ff55f74c94f9794e5d1da8471def8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:38.324421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '729ff55f74c94f9794e5d1da8471def8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:38.325468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '729ff55f74c94f9794e5d1da8471def8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:38.344173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '729ff55f74c94f9794e5d1da8471def8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:38.351263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1237/2518] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:38 2022] DELETE /identity/v3/users/729ff55f74c94f9794e5d1da8471def8 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:38.359657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:38.360299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:38.361562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:38.393015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:38.395010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:38.395316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:38.396801 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:38.421294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:38.467284 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38ad0b6-ded7-475c-ab9e-3a5fc742f74b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:38.471776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:38.472456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:38.472748 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:38.473051 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:38.475304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:38.507146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1282/2519] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:38.518062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:38.520241 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:38.528992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:38.566019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:38.567548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:38.579367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1238/2520] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:38 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:47:38.605410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece8c7c-e2eb-466e-bb11-c30862ada47f None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:38.609350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:38.610020 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:38.610478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:38.610854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:38.622103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:38.623642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:38.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1283/2521] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:39.119276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:39.121685 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:39.151949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d5900-3eca-45ce-9466-ba2e53c34338 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:39.160713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:39.162288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:39.162826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:39.163327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] PATH_INFO: `/v3/projects/04e89378a1734f6c881e0e6234986a6b` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:39.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04e89378a1734f6c881e0e6234986a6b)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:39.169793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:39.264599 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04e89378a1734f6c881e0e6234986a6b', 'request_id': 'req-c86044d7-b370-4235-9883-e4d287cff0f2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.276014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04e89378a1734f6c881e0e6234986a6b', 'request_id': 'req-c86044d7-b370-4235-9883-e4d287cff0f2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.289863 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04e89378a1734f6c881e0e6234986a6b', 'request_id': 'req-c86044d7-b370-4235-9883-e4d287cff0f2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.305307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] The token cache is being invalidate because project 04e89378a1734f6c881e0e6234986a6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:39.305759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.306609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.308527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1239/2522] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:39 2022] DELETE /identity/v3/projects/04e89378a1734f6c881e0e6234986a6b => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:39.313764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:39.327344 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf902dac-5064-4964-beab-af24e5c28dda service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:39.396647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:39.397570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:39.397866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:39.398143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] PATH_INFO: `/v3/users/4b016045e1584f3c9ac7608965ce3b12` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:39.399524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b016045e1584f3c9ac7608965ce3b12)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:39.401522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:39.465050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b016045e1584f3c9ac7608965ce3b12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.473966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b016045e1584f3c9ac7608965ce3b12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.474840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b016045e1584f3c9ac7608965ce3b12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.484187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b016045e1584f3c9ac7608965ce3b12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.506917 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-09070267-d31d-4198-a858-46a5d49e0b51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b016045e1584f3c9ac7608965ce3b12'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:39.518898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1284/2523] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:39 2022] DELETE /identity/v3/users/4b016045e1584f3c9ac7608965ce3b12 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:39.879790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:39.882654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:39.914102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86044d7-b370-4235-9883-e4d287cff0f2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:39.980943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:39.994715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:39.995792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:39.996113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:39.996710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] PATH_INFO: `/v3/projects/85f7503b5c774d39b1ce03f364e9104f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:39.998288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=85f7503b5c774d39b1ce03f364e9104f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.000325 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.057039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85f7503b5c774d39b1ce03f364e9104f', 'request_id': 'req-f8d2b055-be49-4926-9bf7-bde26fadd0b3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:40.063497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85f7503b5c774d39b1ce03f364e9104f', 'request_id': 'req-f8d2b055-be49-4926-9bf7-bde26fadd0b3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:40.072217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85f7503b5c774d39b1ce03f364e9104f', 'request_id': 'req-f8d2b055-be49-4926-9bf7-bde26fadd0b3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:40.086722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] The token cache is being invalidate because project 85f7503b5c774d39b1ce03f364e9104f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:40.087065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:40.087977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-f8d2b055-be49-4926-9bf7-bde26fadd0b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:40.089848 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1240/2524] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:39 2022] DELETE /identity/v3/projects/85f7503b5c774d39b1ce03f364e9104f => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:40.096021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8459f7-6709-4505-89b0-b756a59387e7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.096367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8459f7-6709-4505-89b0-b756a59387e7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.096713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8459f7-6709-4505-89b0-b756a59387e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.117455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5a8459f7-6709-4505-89b0-b756a59387e7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:40.118650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5a8459f7-6709-4505-89b0-b756a59387e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:40.145698 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:40.205079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1285/2525] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:40.213336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cdb40-140a-4f26-9931-5daee617e3d3 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.214026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cdb40-140a-4f26-9931-5daee617e3d3 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.214026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cdb40-140a-4f26-9931-5daee617e3d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.233946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-eb6cdb40-140a-4f26-9931-5daee617e3d3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:40.235234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-eb6cdb40-140a-4f26-9931-5daee617e3d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:40.244359 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1241/2526] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:40.249703 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8459f7-6709-4505-89b0-b756a59387e7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.268296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8459f7-6709-4505-89b0-b756a59387e7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.275580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.276393 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.276857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.277159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.278506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:40.279079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.282177 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.298204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1286/2527] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:40.302179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6cdb40-140a-4f26-9931-5daee617e3d3 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.312305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6cdb40-140a-4f26-9931-5daee617e3d3 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.317839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.318639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.319022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.319368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.320117 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.322191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.352984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1242/2528] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:40 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:40.359187 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.370319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d797eca-f95f-40b6-9888-764074982dc1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.455041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.455964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.456243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.456664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.457867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.460158 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.488387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1287/2529] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:40 2022] POST /identity/v3/users => generated 293 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:40.492564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.503950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecf57c4-77da-4c97-bde9-028eeb0f6edc admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.509052 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.509755 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.510019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.510314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.511111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.512769 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.525788 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1243/2530] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:40.531669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.543390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cdffd5-a47b-404d-ae58-7bc2aa5b8dfe admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.548441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa422b53-401b-4fa4-a380-1fc2e633c672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.549132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa422b53-401b-4fa4-a380-1fc2e633c672 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.549418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa422b53-401b-4fa4-a380-1fc2e633c672 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.549694 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa422b53-401b-4fa4-a380-1fc2e633c672 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.550470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa422b53-401b-4fa4-a380-1fc2e633c672 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.552152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa422b53-401b-4fa4-a380-1fc2e633c672 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.563965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1288/2531] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:40.568550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.580488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761fee31-694e-424a-bd95-a0b6c3f05a8f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.586254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.587206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.587493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.587787 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] PATH_INFO: `/v3/projects/a159e3b5a9d04537bdb2ff2bb379bbd5/users/5a314f4b8376454fba0de26bac18bc61/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.605982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a159e3b5a9d04537bdb2ff2bb379bbd5, user_id=5a314f4b8376454fba0de26bac18bc61, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.609185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.625332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1244/2532] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:40 2022] PUT /identity/v3/projects/a159e3b5a9d04537bdb2ff2bb379bbd5/users/5a314f4b8376454fba0de26bac18bc61/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:40.638465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a6b44-4c01-4b08-814a-62ad5c94b0f7 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.638780 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a6b44-4c01-4b08-814a-62ad5c94b0f7 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.639065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a6b44-4c01-4b08-814a-62ad5c94b0f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.671786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-190a6b44-4c01-4b08-814a-62ad5c94b0f7 None None] MFA Rules not processed for user `5a314f4b8376454fba0de26bac18bc61`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:40.673091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-190a6b44-4c01-4b08-814a-62ad5c94b0f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:40.754725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1289/2533] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:40 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:40.761354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.837419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36fcff1-7655-4d97-8f25-7e5afe9b9a13 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.843306 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.844005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.844288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.844564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.845278 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.847336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:40.881203 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1245/2534] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:40 2022] POST /identity/v3/projects => generated 387 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:40.882729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a6b44-4c01-4b08-814a-62ad5c94b0f7 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:40.896038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a6b44-4c01-4b08-814a-62ad5c94b0f7 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:40.918697 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:40.976773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:40.978201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:40.978989 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:40.979470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:40.980877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:40.984778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:41.014739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1290/2535] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:40 2022] POST /identity/v3/users => generated 300 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:41.021196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:41.028722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:41.035123 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6065f3-7920-4953-84c1-dca1fd84250b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:41.044099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:41.044807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.045380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.045738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.046675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:41.049109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:41.061248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1246/2536] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:41 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:41.066294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:41.077322 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a075e9f3-2cc9-4e22-b398-20f1f2187a3e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:41.083240 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:41.084417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.084894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.085284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] PATH_INFO: `/v3/projects/bd13babe04ab4a949cdb7153870872cf/users/839f5585a1864a94b8fb7ed11b15676e/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.103996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd13babe04ab4a949cdb7153870872cf, user_id=839f5585a1864a94b8fb7ed11b15676e, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:41.106743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:41.122604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1291/2537] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:41 2022] PUT /identity/v3/projects/bd13babe04ab4a949cdb7153870872cf/users/839f5585a1864a94b8fb7ed11b15676e/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:41.129938 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:41.148317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e5f1d-54a5-4617-8709-dd2c4441efa3 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:41.252700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9e5f15-6132-467f-be70-cfe2c1b7c8b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:41.253810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e5f15-6132-467f-be70-cfe2c1b7c8b3 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.254231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e5f15-6132-467f-be70-cfe2c1b7c8b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.254752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e5f15-6132-467f-be70-cfe2c1b7c8b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.256031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9e5f15-6132-467f-be70-cfe2c1b7c8b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:41.258954 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9e5f15-6132-467f-be70-cfe2c1b7c8b3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:41.276747 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1247/2538] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:41 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:41.281832 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:41.293264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ab4d76-c9e0-4898-9273-af8a25592a1a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:41.298669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:41.299590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.299874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.300151 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] PATH_INFO: `/v3/projects/bd13babe04ab4a949cdb7153870872cf/users/839f5585a1864a94b8fb7ed11b15676e/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.303341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd13babe04ab4a949cdb7153870872cf, user_id=839f5585a1864a94b8fb7ed11b15676e, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:41.309095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:41.329392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1292/2539] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:41 2022] PUT /identity/v3/projects/bd13babe04ab4a949cdb7153870872cf/users/839f5585a1864a94b8fb7ed11b15676e/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:41.340625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756fcef-1725-47e1-8179-516e088e82c5 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.340905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756fcef-1725-47e1-8179-516e088e82c5 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.341188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756fcef-1725-47e1-8179-516e088e82c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.379608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3756fcef-1725-47e1-8179-516e088e82c5 None None] MFA Rules not processed for user `839f5585a1864a94b8fb7ed11b15676e`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:41.381657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3756fcef-1725-47e1-8179-516e088e82c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:41.512741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1248/2540] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:41 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:41.527869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:41.529965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:41.609868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420ebfeb-42c4-437b-aa6c-1511845c1850 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:41.614176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ead40c-4e00-41cb-91f1-4e86ea9c2667 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:41.615100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ead40c-4e00-41cb-91f1-4e86ea9c2667 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.615594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ead40c-4e00-41cb-91f1-4e86ea9c2667 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.617289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ead40c-4e00-41cb-91f1-4e86ea9c2667 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.627079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ead40c-4e00-41cb-91f1-4e86ea9c2667 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:41.628568 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ead40c-4e00-41cb-91f1-4e86ea9c2667 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:41.640404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1293/2541] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:41 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:41.674190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756fcef-1725-47e1-8179-516e088e82c5 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:41.748129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756fcef-1725-47e1-8179-516e088e82c5 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:41.753612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:41.754641 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.754970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.755311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] PATH_INFO: `/v3/users/5a314f4b8376454fba0de26bac18bc61` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.756635 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a314f4b8376454fba0de26bac18bc61)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:41.759005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:41.779639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:41.818330 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a314f4b8376454fba0de26bac18bc61'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:41.826682 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a314f4b8376454fba0de26bac18bc61'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:41.836957 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a314f4b8376454fba0de26bac18bc61'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:41.837625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a314f4b8376454fba0de26bac18bc61'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:41.852590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a314f4b8376454fba0de26bac18bc61'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:41.859368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1249/2542] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:41 2022] DELETE /identity/v3/users/5a314f4b8376454fba0de26bac18bc61 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:41.864959 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:41.865255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:41.865708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:41.885891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:41.887180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:41.956715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1294/2543] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:41.967678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:41.969194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:42.054111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea363012-2320-4e32-af1c-ec4b13a56a4a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:42.057911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:42.058689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:42.058973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:42.059251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:42.070355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:42.071860 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:42.083226 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1250/2544] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:42.447441 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:42.449640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:42.466761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5401be1-d70e-4c6e-be8c-581db67e4e3b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:42.472726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:42.473738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:42.474084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:42.474562 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] PATH_INFO: `/v3/projects/a159e3b5a9d04537bdb2ff2bb379bbd5` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:42.475975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a159e3b5a9d04537bdb2ff2bb379bbd5)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:42.477996 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:42.497229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:42.545962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a159e3b5a9d04537bdb2ff2bb379bbd5', 'request_id': 'req-959fb000-a491-492b-9917-1eca1439d23e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.555979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a159e3b5a9d04537bdb2ff2bb379bbd5', 'request_id': 'req-959fb000-a491-492b-9917-1eca1439d23e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.564107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a159e3b5a9d04537bdb2ff2bb379bbd5', 'request_id': 'req-959fb000-a491-492b-9917-1eca1439d23e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.582965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] The token cache is being invalidate because project a159e3b5a9d04537bdb2ff2bb379bbd5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:42.583402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.584449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.586840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1295/2545] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:42 2022] DELETE /identity/v3/projects/a159e3b5a9d04537bdb2ff2bb379bbd5 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:42.591721 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:42.606544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bff9cc-0af8-46be-a8bf-cbfb6858ff34 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:42.681481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:42.682405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:42.682732 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:42.683010 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] PATH_INFO: `/v3/users/839f5585a1864a94b8fb7ed11b15676e` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:42.684360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=839f5585a1864a94b8fb7ed11b15676e)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:42.686415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:42.738272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:42.749067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '839f5585a1864a94b8fb7ed11b15676e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.761280 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '839f5585a1864a94b8fb7ed11b15676e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.773162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '839f5585a1864a94b8fb7ed11b15676e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.774164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '839f5585a1864a94b8fb7ed11b15676e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.795372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2250da9d-de55-47e8-92e4-99bff05a01f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '839f5585a1864a94b8fb7ed11b15676e'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:42.803767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1251/2546] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:42 2022] DELETE /identity/v3/users/839f5585a1864a94b8fb7ed11b15676e => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:43.133716 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:43.135369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:43.153935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959fb000-a491-492b-9917-1eca1439d23e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:43.228876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:43.229827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.230117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.230406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] PATH_INFO: `/v3/projects/bd13babe04ab4a949cdb7153870872cf` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.232526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd13babe04ab4a949cdb7153870872cf)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:43.236493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:43.296965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd13babe04ab4a949cdb7153870872cf', 'request_id': 'req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:43.306380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd13babe04ab4a949cdb7153870872cf', 'request_id': 'req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:43.313112 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd13babe04ab4a949cdb7153870872cf', 'request_id': 'req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:43.326511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] The token cache is being invalidate because project bd13babe04ab4a949cdb7153870872cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:43.326827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:43.327542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-75fe9039-5f6e-406f-a1c1-ccc4c2e1319e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:43.329756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1296/2547] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:43 2022] DELETE /identity/v3/projects/bd13babe04ab4a949cdb7153870872cf => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:43.335895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3a9bc6-2a0f-4d11-b3a3-08066113969e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.336255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3a9bc6-2a0f-4d11-b3a3-08066113969e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.336510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3a9bc6-2a0f-4d11-b3a3-08066113969e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.360493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cc3a9bc6-2a0f-4d11-b3a3-08066113969e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:43.361031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cc3a9bc6-2a0f-4d11-b3a3-08066113969e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:43.465335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1252/2548] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:43.474701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b993a4-48e6-4b94-ac4a-04287cb2b1cc None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.475066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b993a4-48e6-4b94-ac4a-04287cb2b1cc None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.475066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b993a4-48e6-4b94-ac4a-04287cb2b1cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.498606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e3b993a4-48e6-4b94-ac4a-04287cb2b1cc None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:43.499889 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e3b993a4-48e6-4b94-ac4a-04287cb2b1cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:43.508343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1297/2549] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:43.513853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3a9bc6-2a0f-4d11-b3a3-08066113969e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:43.532198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3a9bc6-2a0f-4d11-b3a3-08066113969e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:43.541428 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:43.542705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.543211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.543793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.545210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:43.545864 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:43.549513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:43.571000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1253/2550] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:43.575958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b993a4-48e6-4b94-ac4a-04287cb2b1cc None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:43.587756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b993a4-48e6-4b94-ac4a-04287cb2b1cc None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:43.593196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:43.594027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.594422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.595479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.596669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:43.599118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:43.632810 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1298/2551] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:43 2022] POST /identity/v3/projects => generated 387 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:43.637194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:43.655859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2c0a96-aeb0-4257-aa60-3d4f87814170 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:43.748229 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:43.749254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.749597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.749923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.750760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:43.753517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:43.778356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1254/2552] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:43 2022] POST /identity/v3/users => generated 294 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:43.783474 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:43.789746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:43.796717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b952241-41c5-4df5-8c1e-024bea6cfca2 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:43.804117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:43.804993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.805332 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.805752 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.806712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:43.808734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:43.951444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1299/2553] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:43 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:47:43.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:43.969628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6159ff8-85aa-4be0-8f05-b50490f7f7a8 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:43.977888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f69cf5-989a-4b5a-a50f-5d0a31bd2a94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:43.979139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f69cf5-989a-4b5a-a50f-5d0a31bd2a94 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:43.979626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f69cf5-989a-4b5a-a50f-5d0a31bd2a94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:43.980104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f69cf5-989a-4b5a-a50f-5d0a31bd2a94 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:43.981495 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f69cf5-989a-4b5a-a50f-5d0a31bd2a94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:43.984506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f69cf5-989a-4b5a-a50f-5d0a31bd2a94 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.006750 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1255/2554] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:43 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:47:44.011576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:44.023948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf501a52-64af-4657-a336-6f496366e31b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:44.028857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:44.030075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.030355 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.030671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] PATH_INFO: `/v3/projects/c34eea795dfb4e01baf737993fb4a140/users/d562028d12b041a0842ed5242fa54269/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.049091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c34eea795dfb4e01baf737993fb4a140, user_id=d562028d12b041a0842ed5242fa54269, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:44.051971 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.068210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1300/2555] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:44 2022] PUT /identity/v3/projects/c34eea795dfb4e01baf737993fb4a140/users/d562028d12b041a0842ed5242fa54269/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:44.080062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093b63c-ade2-4a37-b077-ea9503e30120 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.080381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093b63c-ade2-4a37-b077-ea9503e30120 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.080627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093b63c-ade2-4a37-b077-ea9503e30120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.111670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2093b63c-ade2-4a37-b077-ea9503e30120 None None] MFA Rules not processed for user `d562028d12b041a0842ed5242fa54269`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:44.113102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2093b63c-ade2-4a37-b077-ea9503e30120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:44.123722 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:44.213878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1256/2556] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:44.219812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:44.263453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a6b92-be91-4a1b-82f7-b4af2c55c894 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.264100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a6b92-be91-4a1b-82f7-b4af2c55c894 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.264649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a6b92-be91-4a1b-82f7-b4af2c55c894 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.298949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fc4a6b92-be91-4a1b-82f7-b4af2c55c894 None None] MFA Rules not processed for user `0f8e300cd96e446db7430044e69d997a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:44.301150 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fc4a6b92-be91-4a1b-82f7-b4af2c55c894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:44.337134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7866555a-f3cd-4a5e-b5e3-278793b7f400 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:44.345304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:44.346597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.347299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.347835 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.349155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:44.352405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.401820 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1257/2557] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:44 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:44.406638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1301/2558] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:44 2022] POST /identity/v3/projects => generated 385 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:44.411767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a6b92-be91-4a1b-82f7-b4af2c55c894 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:44.431888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a6b92-be91-4a1b-82f7-b4af2c55c894 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:44.572100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:44.574197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.575320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.576009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.577896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:44.583680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.630472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1258/2559] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:44 2022] POST /identity/v3/users => generated 299 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:44.634372 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:44.647818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015ed03d-c140-43e5-a1e5-a24b4ec87eb0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:44.656648 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:44.658539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.659239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.659943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.661138 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:44.663017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.683914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1302/2560] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:44 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:44.689220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:44.706485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a6d4be-61c7-4d92-b6e5-238cabf739f1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:44.712890 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:44.714153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.714617 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.714964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] PATH_INFO: `/v3/projects/521b1b261bbc4180a2ba36e2ef64c772/users/3db699752f17476e806b0c6d7b72e814/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.736396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=521b1b261bbc4180a2ba36e2ef64c772, user_id=3db699752f17476e806b0c6d7b72e814, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:44.739410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.756776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1259/2561] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:44 2022] PUT /identity/v3/projects/521b1b261bbc4180a2ba36e2ef64c772/users/3db699752f17476e806b0c6d7b72e814/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:44.761107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:44.773363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7b2be-c3c4-48c5-9a05-4af1cc8b67ae admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:44.869084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39d8561-e3ed-4422-af5a-2cb294e2ba94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:44.870286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39d8561-e3ed-4422-af5a-2cb294e2ba94 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.870877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39d8561-e3ed-4422-af5a-2cb294e2ba94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.871630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39d8561-e3ed-4422-af5a-2cb294e2ba94 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.872730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39d8561-e3ed-4422-af5a-2cb294e2ba94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:44.875872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39d8561-e3ed-4422-af5a-2cb294e2ba94 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.896943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1303/2562] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:44 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:44.903398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:44.925811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88f3d21-3ab1-4362-898c-8745bc003d43 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:44.933381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:44.934313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.934650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.934937 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] PATH_INFO: `/v3/projects/521b1b261bbc4180a2ba36e2ef64c772/users/3db699752f17476e806b0c6d7b72e814/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.937154 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] RBAC: Authorizing `identity:create_grant(project_id=521b1b261bbc4180a2ba36e2ef64c772, user_id=3db699752f17476e806b0c6d7b72e814, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:44.940089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:44.952961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1260/2563] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:44 2022] PUT /identity/v3/projects/521b1b261bbc4180a2ba36e2ef64c772/users/3db699752f17476e806b0c6d7b72e814/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:44.964630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a6fa90-34e6-4373-824b-24df105d2b92 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:44.965065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a6fa90-34e6-4373-824b-24df105d2b92 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:44.965461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a6fa90-34e6-4373-824b-24df105d2b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:44.997553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a1a6fa90-34e6-4373-824b-24df105d2b92 None None] MFA Rules not processed for user `3db699752f17476e806b0c6d7b72e814`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:44.999426 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a1a6fa90-34e6-4373-824b-24df105d2b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:45.031914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:45.099359 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1304/2564] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:44 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:45.112313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:45.114476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:45.128588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:45.186943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b636fcd-2f0e-4b7b-9dfc-2bc28bd9e37b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:45.191521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617dfc76-f083-4676-932a-bda404b3d456 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:45.192200 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617dfc76-f083-4676-932a-bda404b3d456 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:45.192478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617dfc76-f083-4676-932a-bda404b3d456 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:45.192756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617dfc76-f083-4676-932a-bda404b3d456 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:45.208513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617dfc76-f083-4676-932a-bda404b3d456 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:45.210301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617dfc76-f083-4676-932a-bda404b3d456 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:45.222827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1261/2565] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:45 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:45.316824 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a6fa90-34e6-4373-824b-24df105d2b92 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:45.432586 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a6fa90-34e6-4373-824b-24df105d2b92 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:45.446469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:45.447627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:45.448198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:45.448812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] PATH_INFO: `/v3/users/d562028d12b041a0842ed5242fa54269` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:45.451080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d562028d12b041a0842ed5242fa54269)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:45.455313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:45.512995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd562028d12b041a0842ed5242fa54269'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.524524 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd562028d12b041a0842ed5242fa54269'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.525234 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd562028d12b041a0842ed5242fa54269'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.537375 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd562028d12b041a0842ed5242fa54269'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.564287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd562028d12b041a0842ed5242fa54269'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.572229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1305/2566] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:45 2022] DELETE /identity/v3/users/d562028d12b041a0842ed5242fa54269 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:45.577648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:45.577648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:45.577648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:45.598188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:45.600272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:45.715821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1262/2567] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:45.726739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:45.728922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:45.739382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:45.744195 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:45.766540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d090d9fb-7b6b-4edd-ae71-83bd183f13d1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:45.774712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:45.775760 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:45.776040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:45.776503 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] PATH_INFO: `/v3/users/ab9b328a88294277a7fe957c18488c6f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:45.779229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab9b328a88294277a7fe957c18488c6f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:45.782055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:45.853224 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e48f8d2-5145-406c-b5b4-152dd1d4f503 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:45.861081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c9fa7c-9906-4ca8-b6a0-2d30c19f8386 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:45.862196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c9fa7c-9906-4ca8-b6a0-2d30c19f8386 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:45.862525 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c9fa7c-9906-4ca8-b6a0-2d30c19f8386 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:45.862809 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c9fa7c-9906-4ca8-b6a0-2d30c19f8386 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:45.879014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c9fa7c-9906-4ca8-b6a0-2d30c19f8386 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:45.879513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:45.880507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c9fa7c-9906-4ca8-b6a0-2d30c19f8386 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:45.901329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab9b328a88294277a7fe957c18488c6f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.906196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1263/2568] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:45.917560 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab9b328a88294277a7fe957c18488c6f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.918711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab9b328a88294277a7fe957c18488c6f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.943419 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab9b328a88294277a7fe957c18488c6f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:45.979492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab9b328a88294277a7fe957c18488c6f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.001813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1306/2569] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:45 2022] DELETE /identity/v3/users/ab9b328a88294277a7fe957c18488c6f => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:46.009147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:46.009913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:46.010410 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:46.044048 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:46.045485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:46.123815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:46.148601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1264/2570] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:46.166938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:46.169152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:46.281091 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c777e2a6-fae9-4da0-8f58-df38d6576c1b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:46.286871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:46.287642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:46.288035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:46.288328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:46.300744 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:46.302542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:46.320845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1307/2571] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:46.493742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:46.496219 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:46.519203 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfb9da2-79d2-44dd-9a66-1e6bfe7a963c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:46.525603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:46.526796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:46.527874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:46.531056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] PATH_INFO: `/v3/projects/c34eea795dfb4e01baf737993fb4a140` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:46.538990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c34eea795dfb4e01baf737993fb4a140)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:46.545152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:46.643868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c34eea795dfb4e01baf737993fb4a140', 'request_id': 'req-e7bfbc02-3217-4184-b61b-bc1f37a20022'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.651542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c34eea795dfb4e01baf737993fb4a140', 'request_id': 'req-e7bfbc02-3217-4184-b61b-bc1f37a20022'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.663671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c34eea795dfb4e01baf737993fb4a140', 'request_id': 'req-e7bfbc02-3217-4184-b61b-bc1f37a20022'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.679072 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] The token cache is being invalidate because project c34eea795dfb4e01baf737993fb4a140 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:46.679717 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.680522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.682288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1265/2572] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:46 2022] DELETE /identity/v3/projects/c34eea795dfb4e01baf737993fb4a140 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:46.688036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:46.702281 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5318aa3b-91b6-4d19-83f8-d97c1dd90e13 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:46.817828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:46.826843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:46.826843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:46.826843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] PATH_INFO: `/v3/users/3db699752f17476e806b0c6d7b72e814` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:46.826843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3db699752f17476e806b0c6d7b72e814)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:46.826843 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:46.861610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:46.863488 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:46.881254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfbc02-3217-4184-b61b-bc1f37a20022 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:46.887149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:46.888083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:46.888365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:46.888658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] PATH_INFO: `/v3/projects/886e243aa71042c0b11a4444878c600a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:46.890644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=886e243aa71042c0b11a4444878c600a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:46.893570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:46.913179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3db699752f17476e806b0c6d7b72e814'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.922690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3db699752f17476e806b0c6d7b72e814'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.923381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3db699752f17476e806b0c6d7b72e814'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.929350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:46.934712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3db699752f17476e806b0c6d7b72e814'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.956119 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1e47ea53-676c-4036-992c-b3ec0772b0e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3db699752f17476e806b0c6d7b72e814'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.962909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '886e243aa71042c0b11a4444878c600a', 'request_id': 'req-2e84b66c-3878-458e-8403-f7d749e60b62'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.963455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1308/2573] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:47:46 2022] DELETE /identity/v3/users/3db699752f17476e806b0c6d7b72e814 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:46.969166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '886e243aa71042c0b11a4444878c600a', 'request_id': 'req-2e84b66c-3878-458e-8403-f7d749e60b62'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.980710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '886e243aa71042c0b11a4444878c600a', 'request_id': 'req-2e84b66c-3878-458e-8403-f7d749e60b62'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.998355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] The token cache is being invalidate because project 886e243aa71042c0b11a4444878c600a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:46.998772 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:46.999367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-2e84b66c-3878-458e-8403-f7d749e60b62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:47.002521 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1266/2574] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:46 2022] DELETE /identity/v3/projects/886e243aa71042c0b11a4444878c600a => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:47:47.009082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b510ca-ad66-4d0f-baa5-08d000648ea4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.009082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b510ca-ad66-4d0f-baa5-08d000648ea4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.009082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b510ca-ad66-4d0f-baa5-08d000648ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.036095 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e9b510ca-ad66-4d0f-baa5-08d000648ea4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:47.038992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e9b510ca-ad66-4d0f-baa5-08d000648ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.130597 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1309/2575] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:47.140640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aacdab-8cb2-4d6c-9472-d3f4d2b9ba32 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.141029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aacdab-8cb2-4d6c-9472-d3f4d2b9ba32 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.141208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aacdab-8cb2-4d6c-9472-d3f4d2b9ba32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.170582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-e9aacdab-8cb2-4d6c-9472-d3f4d2b9ba32 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:47.171961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e9aacdab-8cb2-4d6c-9472-d3f4d2b9ba32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.181494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1267/2576] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:47:47.186706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b510ca-ad66-4d0f-baa5-08d000648ea4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.205743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b510ca-ad66-4d0f-baa5-08d000648ea4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.214668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.215915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.216492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.217129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.218531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:47.218927 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.221386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.245466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1310/2577] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:47:47.251822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9aacdab-8cb2-4d6c-9472-d3f4d2b9ba32 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.270850 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9aacdab-8cb2-4d6c-9472-d3f4d2b9ba32 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.279911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.281120 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.281710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.282336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.283932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.288133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.309353 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.312559 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.330414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1268/2578] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/projects => generated 391 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:47.335821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.345370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaaa72a-944a-4d8c-9765-14c284ba6003 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.390327 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:47.428110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.428933 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1138b0-e0b0-4b44-8564-e2ff66b547d6 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.429217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.429604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.429986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] PATH_INFO: `/v3/projects/521b1b261bbc4180a2ba36e2ef64c772` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.431255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] RBAC: Authorizing `identity:delete_project(project_id=521b1b261bbc4180a2ba36e2ef64c772)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.433227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.434906 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.435744 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.436105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.436481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.437314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.439658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.473128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1269/2579] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/users => generated 296 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:47.475050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.492068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21c453-d37b-460e-bef1-2e79681d3b5d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.497679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '521b1b261bbc4180a2ba36e2ef64c772', 'request_id': 'req-80fb476b-7bd9-445b-affb-5809cb8c5922'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:47.507381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '521b1b261bbc4180a2ba36e2ef64c772', 'request_id': 'req-80fb476b-7bd9-445b-affb-5809cb8c5922'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:47.514047 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '521b1b261bbc4180a2ba36e2ef64c772', 'request_id': 'req-80fb476b-7bd9-445b-affb-5809cb8c5922'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:47.515388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:47.533553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] The token cache is being invalidate because project 521b1b261bbc4180a2ba36e2ef64c772 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:47:47.534149 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:47.535159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-80fb476b-7bd9-445b-affb-5809cb8c5922 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:47:47.537036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1311/2580] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:47:47 2022] DELETE /identity/v3/projects/521b1b261bbc4180a2ba36e2ef64c772 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:47.544099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.544610 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.545122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.566724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9802769e-5cdd-42ab-8e61-73ac85f78338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.567794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9802769e-5cdd-42ab-8e61-73ac85f78338 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.568264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9802769e-5cdd-42ab-8e61-73ac85f78338 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.568692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9802769e-5cdd-42ab-8e61-73ac85f78338 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.569593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9802769e-5cdd-42ab-8e61-73ac85f78338 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.571572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9802769e-5cdd-42ab-8e61-73ac85f78338 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.576019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:47.577346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.586578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1312/2581] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:47.589469 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1270/2582] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:47 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:47:47.591075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.592756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.595472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc295e3c-ec0d-46e7-a23c-ed72db698790 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.595763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc295e3c-ec0d-46e7-a23c-ed72db698790 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.596048 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc295e3c-ec0d-46e7-a23c-ed72db698790 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.607929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d01c4f-426c-410d-b2b0-3cbfe31687b8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.613914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.614809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.615206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.615682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.616335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fc295e3c-ec0d-46e7-a23c-ed72db698790 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:47.617136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.617602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fc295e3c-ec0d-46e7-a23c-ed72db698790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.619262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.627940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1271/2583] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:47.637369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1313/2584] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:47 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:47:47.637652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc295e3c-ec0d-46e7-a23c-ed72db698790 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.639657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.645254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc295e3c-ec0d-46e7-a23c-ed72db698790 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.651973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.652319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd051166-acef-494f-bd3b-88b2826c04f6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.652916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.653296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.653627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.654735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:47:47.655259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.664711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.665048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d016f5ef-3323-4070-9834-2251db2e72c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.665048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016f5ef-3323-4070-9834-2251db2e72c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.665453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016f5ef-3323-4070-9834-2251db2e72c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.665958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016f5ef-3323-4070-9834-2251db2e72c4 admin admin] PATH_INFO: `/v3/projects/94742fdcd5e14980b5055367ad97611f/users/6c255c066ed841dab1af25a45fb669b9/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.679080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1272/2585] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:47:47.684230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.689727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016f5ef-3323-4070-9834-2251db2e72c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94742fdcd5e14980b5055367ad97611f, user_id=6c255c066ed841dab1af25a45fb669b9, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.693415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016f5ef-3323-4070-9834-2251db2e72c4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.695135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da17d4c-ed25-44f6-946c-2194d8ee076f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.700404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.701098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.701378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.701661 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.702380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.704430 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.712256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1314/2586] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:47 2022] PUT /identity/v3/projects/94742fdcd5e14980b5055367ad97611f/users/6c255c066ed841dab1af25a45fb669b9/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:47.732301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20feb65-cdce-437e-9561-e634d5b5c1a8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.732604 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20feb65-cdce-437e-9561-e634d5b5c1a8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.732872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20feb65-cdce-437e-9561-e634d5b5c1a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.743332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1273/2587] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/projects => generated 377 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:47:47.746785 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.771024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c20feb65-cdce-437e-9561-e634d5b5c1a8 None None] MFA Rules not processed for user `6c255c066ed841dab1af25a45fb669b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:47.772815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c20feb65-cdce-437e-9561-e634d5b5c1a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.833194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa21d813-b31f-4b00-9ca8-e875c79171a2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.842246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.843967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.844296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.845531 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.847093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.849662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.884672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1315/2588] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:47.892145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1274/2589] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:47:47 2022] POST /identity/v3/users => generated 289 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:47:47.897138 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20feb65-cdce-437e-9561-e634d5b5c1a8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.899440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.902031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:47.911257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20feb65-cdce-437e-9561-e634d5b5c1a8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.917399 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.918245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.918668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.919065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.919984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.920436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:47.921945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.942805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1316/2590] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:47 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:47:47.947625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:47.965460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1351c2-594b-43ba-b87c-58abacbfbcac admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:47.972706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:47.973735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:47.974219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:47.974756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:47.975924 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:47.978726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:47.991401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1317/2591] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:47:47 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:47:47.996009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:48.010073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f0fb29-ac03-471c-901d-f897e238c3a1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:48.011167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3e57b-d872-4120-bb1b-b3811caa01da admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:48.015378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ecfd24-f3d3-446b-976e-201f5ec2ae91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:48.016198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ecfd24-f3d3-446b-976e-201f5ec2ae91 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:48.017037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ecfd24-f3d3-446b-976e-201f5ec2ae91 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:48.017399 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ecfd24-f3d3-446b-976e-201f5ec2ae91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:48.019084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:48.020632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:48.021146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:48.021649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] PATH_INFO: `/v3/projects/75a17b8d19514c788f2e60fe728b4ca4/users/e298631534ec44aaa2bda10a9a848b10/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:48.029681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ecfd24-f3d3-446b-976e-201f5ec2ae91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:48.031970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ecfd24-f3d3-446b-976e-201f5ec2ae91 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:48.043776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1275/2592] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:47 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:47:48.049543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75a17b8d19514c788f2e60fe728b4ca4, user_id=e298631534ec44aaa2bda10a9a848b10, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:48.054535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:48.077774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1318/2593] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:47:47 2022] PUT /identity/v3/projects/75a17b8d19514c788f2e60fe728b4ca4/users/e298631534ec44aaa2bda10a9a848b10/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:47:48.093202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fade03fe-e2b7-48c1-bac7-e6f092a91cf6 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:48.093487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fade03fe-e2b7-48c1-bac7-e6f092a91cf6 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:48.093961 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fade03fe-e2b7-48c1-bac7-e6f092a91cf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:48.127228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fade03fe-e2b7-48c1-bac7-e6f092a91cf6 None None] MFA Rules not processed for user `e298631534ec44aaa2bda10a9a848b10`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:47:48.128644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fade03fe-e2b7-48c1-bac7-e6f092a91cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:48.210626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1276/2594] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:47:48 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:47:48.224873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:47:48.227009 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:47:48.317611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a57c654-8102-4b7b-9e37-a488316653d0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:47:48.324186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996d5038-f517-450b-94a8-7cd0e07e207e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:47:48.326391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996d5038-f517-450b-94a8-7cd0e07e207e service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:47:48.326923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996d5038-f517-450b-94a8-7cd0e07e207e service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:47:48.328180 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996d5038-f517-450b-94a8-7cd0e07e207e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:47:48.344186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996d5038-f517-450b-94a8-7cd0e07e207e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:47:48.346892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996d5038-f517-450b-94a8-7cd0e07e207e service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:47:48.353692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:48.368955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1319/2595] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:47:48 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:47:58.206524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=55 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:47:58.368285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:16.725731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fade03fe-e2b7-48c1-bac7-e6f092a91cf6 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:16.895690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fade03fe-e2b7-48c1-bac7-e6f092a91cf6 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:16.904121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:16.905639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:16.906158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:16.907319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] PATH_INFO: `/v3/users/6c255c066ed841dab1af25a45fb669b9` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:16.909537 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c255c066ed841dab1af25a45fb669b9)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:16.913343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:17.002538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c255c066ed841dab1af25a45fb669b9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.011277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c255c066ed841dab1af25a45fb669b9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.020356 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c255c066ed841dab1af25a45fb669b9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.020988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c255c066ed841dab1af25a45fb669b9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.035283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c255c066ed841dab1af25a45fb669b9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.042151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1277/2596] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:16 2022] DELETE /identity/v3/users/6c255c066ed841dab1af25a45fb669b9 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:17.047556 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:17.048145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:17.048145 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:17.105082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:17.106334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:17.199733 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1320/2597] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:17.211411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:17.214143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:17.280146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:17.326143 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf70914-2a79-47f9-98e8-1c05ae2e2a71 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:17.330671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:17.331271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:17.331563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:17.331845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:17.347502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:17.349994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:17.365426 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1278/2598] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:48:17.728253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:17.730067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:17.750817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40db0c2-f3a5-4a55-9ff6-ed069bb3d708 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:17.759401 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:17.760563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:17.760970 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:17.761386 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] PATH_INFO: `/v3/projects/94742fdcd5e14980b5055367ad97611f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:17.763425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] RBAC: Authorizing `identity:delete_project(project_id=94742fdcd5e14980b5055367ad97611f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:17.767024 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:17.831957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94742fdcd5e14980b5055367ad97611f', 'request_id': 'req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.846777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94742fdcd5e14980b5055367ad97611f', 'request_id': 'req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.857219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94742fdcd5e14980b5055367ad97611f', 'request_id': 'req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.873482 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] The token cache is being invalidate because project 94742fdcd5e14980b5055367ad97611f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:17.874010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.875729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-6c1a1b65-1eb4-494d-ad21-cc2bc19c14bd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:17.878483 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1321/2599] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:17 2022] DELETE /identity/v3/projects/94742fdcd5e14980b5055367ad97611f => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:19.347660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:19.349293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:19.363256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523d941-0322-4124-81aa-5632f5ea6c01 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:19.456243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:19.457270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:19.457568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:19.458138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] PATH_INFO: `/v3/users/35c5e0aac19d4a6dbdb872e738b222db` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:19.459496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35c5e0aac19d4a6dbdb872e738b222db)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:19.461494 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:19.521503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35c5e0aac19d4a6dbdb872e738b222db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:19.536715 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35c5e0aac19d4a6dbdb872e738b222db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:19.550321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35c5e0aac19d4a6dbdb872e738b222db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:19.552432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35c5e0aac19d4a6dbdb872e738b222db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:19.568985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35c5e0aac19d4a6dbdb872e738b222db'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:19.576648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1279/2600] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:19 2022] DELETE /identity/v3/users/35c5e0aac19d4a6dbdb872e738b222db => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:19.582209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:19.583296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:19.583296 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:19.603504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:19.605082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:19.667143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:19.679424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1322/2601] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:19.690152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:19.692230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:19.753904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:19.774039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594aa718-a2aa-41e8-b82b-b0dd572add92 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:19.778695 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:19.779449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:19.779737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:19.780151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:19.792688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:19.794865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:19.810857 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1280/2602] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:20.365232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:20.367977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:20.386453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54006c-de3e-4428-a35f-31d1b000b7f4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:20.392515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:20.393511 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:20.393877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:20.394157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] PATH_INFO: `/v3/projects/d457ae6b47fb4ab28854c930b3210924` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:20.395607 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d457ae6b47fb4ab28854c930b3210924)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:20.397657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:20.484828 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd457ae6b47fb4ab28854c930b3210924', 'request_id': 'req-2f19fc44-c668-470c-a5c9-d89cc7cbd265'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.503159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd457ae6b47fb4ab28854c930b3210924', 'request_id': 'req-2f19fc44-c668-470c-a5c9-d89cc7cbd265'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.516735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd457ae6b47fb4ab28854c930b3210924', 'request_id': 'req-2f19fc44-c668-470c-a5c9-d89cc7cbd265'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.552121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] The token cache is being invalidate because project d457ae6b47fb4ab28854c930b3210924 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:20.552799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.554539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.557786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1323/2603] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:20 2022] DELETE /identity/v3/projects/d457ae6b47fb4ab28854c930b3210924 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:20.563923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:20.574681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18da3bf-6ce7-4d11-bfcd-bc162c826f04 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:20.691247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:20.693000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:20.694249 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:20.694737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] PATH_INFO: `/v3/users/e10d5e9102d04dcd9c46792c6f20fa4a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:20.697909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e10d5e9102d04dcd9c46792c6f20fa4a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:20.703191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:20.803478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e10d5e9102d04dcd9c46792c6f20fa4a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.814813 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:20.816693 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:20.818473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e10d5e9102d04dcd9c46792c6f20fa4a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.834843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e10d5e9102d04dcd9c46792c6f20fa4a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.836077 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e10d5e9102d04dcd9c46792c6f20fa4a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.864114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e10d5e9102d04dcd9c46792c6f20fa4a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:20.878518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1281/2604] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:20 2022] DELETE /identity/v3/users/e10d5e9102d04dcd9c46792c6f20fa4a => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:20.934032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f19fc44-c668-470c-a5c9-d89cc7cbd265 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:21.013690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:21.014403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:21.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:21.016299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:21.019601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:21.104929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:21.133701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:21.136194 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:21.168214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1324/2605] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:48:21.358244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:21.360779 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:21.398066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:21.399941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:21.422899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e278c0-488f-4c6e-a3ae-2797ad52f0b4 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:21.435277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:21.437963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:21.438248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:21.438567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] PATH_INFO: `/v3/projects/3cda26fef7f74e2ab8e622f278bae40d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:21.440403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cda26fef7f74e2ab8e622f278bae40d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:21.443141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:21.503940 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:21.544914 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b994dfa5-d77d-4c54-9df0-9d6954ffafe6 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:21.555145 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0f2e0a-f275-4b20-9ba5-7f2403236801 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:21.556141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0f2e0a-f275-4b20-9ba5-7f2403236801 service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:21.556509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0f2e0a-f275-4b20-9ba5-7f2403236801 service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:21.557089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0f2e0a-f275-4b20-9ba5-7f2403236801 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:21.559393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-aa0f2e0a-f275-4b20-9ba5-7f2403236801 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:21.570751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cda26fef7f74e2ab8e622f278bae40d', 'request_id': 'req-d257de7d-dd6d-40bb-8c31-effa050907e7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:21.577321 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0f2e0a-f275-4b20-9ba5-7f2403236801 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:21.581256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0f2e0a-f275-4b20-9ba5-7f2403236801 service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:21.591242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cda26fef7f74e2ab8e622f278bae40d', 'request_id': 'req-d257de7d-dd6d-40bb-8c31-effa050907e7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:21.597576 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1282/2606] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:21 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:21.601116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cda26fef7f74e2ab8e622f278bae40d', 'request_id': 'req-d257de7d-dd6d-40bb-8c31-effa050907e7'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:21.614646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] The token cache is being invalidate because project 3cda26fef7f74e2ab8e622f278bae40d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:21.615232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:21.616487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d257de7d-dd6d-40bb-8c31-effa050907e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:21.622236 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1325/2607] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:21 2022] DELETE /identity/v3/projects/3cda26fef7f74e2ab8e622f278bae40d => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:21.624260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9674a47e-f218-4804-b9b1-27fc61ea042a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:21.624758 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9674a47e-f218-4804-b9b1-27fc61ea042a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:21.625977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9674a47e-f218-4804-b9b1-27fc61ea042a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:21.703990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9674a47e-f218-4804-b9b1-27fc61ea042a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:21.705827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9674a47e-f218-4804-b9b1-27fc61ea042a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:21.821065 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1283/2608] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:21.829175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a193ccf0-6106-41be-a5be-83dbfdefef32 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:21.829779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a193ccf0-6106-41be-a5be-83dbfdefef32 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:21.830486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a193ccf0-6106-41be-a5be-83dbfdefef32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:21.858421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a193ccf0-6106-41be-a5be-83dbfdefef32 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:21.862031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a193ccf0-6106-41be-a5be-83dbfdefef32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:21.876720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9674a47e-f218-4804-b9b1-27fc61ea042a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:21.877821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1326/2609] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:21.890319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9674a47e-f218-4804-b9b1-27fc61ea042a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:21.897813 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:21.898455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:21.898763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:21.899046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:21.900021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:21.900337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:21.902633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:21.922492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1284/2610] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:48:21.926891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a193ccf0-6106-41be-a5be-83dbfdefef32 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:21.938642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a193ccf0-6106-41be-a5be-83dbfdefef32 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:21.945564 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:21.946289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:21.946583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:21.946869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:21.947617 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:21.949614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:21.990079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1327/2611] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:21 2022] POST /identity/v3/projects => generated 409 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:21.997393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.017152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50bc72b-dd7a-4e07-bb06-ba873d9c5da5 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.112329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.113032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.113485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.113923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.114822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.116916 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.153692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1285/2612] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:21 2022] POST /identity/v3/users => generated 305 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:22.158458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.171300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc9f014-c596-42d0-94c9-3f21ec8af8c6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.194168 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.194969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.195278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.195574 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.196346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.198202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.219190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1328/2613] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:22.220646 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.233855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c1f7e7-b661-4964-95a2-f1caa2b137ce admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.240810 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9bd321-1cf3-4c68-8035-8a4fa7e41d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.241710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bd321-1cf3-4c68-8035-8a4fa7e41d53 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.242202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bd321-1cf3-4c68-8035-8a4fa7e41d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.242526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bd321-1cf3-4c68-8035-8a4fa7e41d53 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.243301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9bd321-1cf3-4c68-8035-8a4fa7e41d53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.245111 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9bd321-1cf3-4c68-8035-8a4fa7e41d53 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.257836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1286/2614] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:22.262231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.273776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef692bd-25a5-4e15-a42e-7d921267d8c0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.279014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.279929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.280209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.280495 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] PATH_INFO: `/v3/projects/88e75c3700e14be18b49f2aff667d137/users/876de5a9796147318d858794d43dae1a/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.299406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=88e75c3700e14be18b49f2aff667d137, user_id=876de5a9796147318d858794d43dae1a, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.303305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.326364 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1329/2615] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:22 2022] PUT /identity/v3/projects/88e75c3700e14be18b49f2aff667d137/users/876de5a9796147318d858794d43dae1a/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:22.343258 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15d722d-5aa6-479d-b269-d4786d06ef74 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.343699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15d722d-5aa6-479d-b269-d4786d06ef74 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.345591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15d722d-5aa6-479d-b269-d4786d06ef74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.396315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f15d722d-5aa6-479d-b269-d4786d06ef74 None None] MFA Rules not processed for user `876de5a9796147318d858794d43dae1a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:22.398523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f15d722d-5aa6-479d-b269-d4786d06ef74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:22.407881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:22.511587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1287/2616] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:22 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:22.517121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.611371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf3fc0-4478-4e8c-9376-c46f053316bf admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.617016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.618129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.618723 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.619254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.620229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.622540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.636049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:22.661198 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1330/2617] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:22 2022] POST /identity/v3/projects => generated 407 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:22.667432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15d722d-5aa6-479d-b269-d4786d06ef74 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.691109 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15d722d-5aa6-479d-b269-d4786d06ef74 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.809955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.810693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.811643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.811643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.811976 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.814595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.838932 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1288/2618] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:22 2022] POST /identity/v3/users => generated 310 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:22.845144 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.859147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f160bb5c-cdd6-4cde-bd05-c3f6f8a9850b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.867176 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.867868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.868153 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.868428 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.869175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.870976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.884453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1331/2619] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:48:22.889648 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.901491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b14469-efc1-4bbe-8cef-417562f362ce admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:22.907411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:22.908342 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:22.908623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:22.908902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] PATH_INFO: `/v3/projects/7a9f8833ef7d44fa885778b1eb866fe4/users/8f9e0997f1af4477b928b0724deb999d/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:22.926492 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a9f8833ef7d44fa885778b1eb866fe4, user_id=8f9e0997f1af4477b928b0724deb999d, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:22.929286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:22.945156 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1289/2620] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:22 2022] PUT /identity/v3/projects/7a9f8833ef7d44fa885778b1eb866fe4/users/8f9e0997f1af4477b928b0724deb999d/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:22.950890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:22.969311 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d985a67-67b6-45be-a8c1-64bf30471d75 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:23.059652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016f118f-689d-446e-bde1-a96e4c6c3e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:23.061003 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016f118f-689d-446e-bde1-a96e4c6c3e2e admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:23.061522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016f118f-689d-446e-bde1-a96e4c6c3e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:23.062042 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016f118f-689d-446e-bde1-a96e4c6c3e2e admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:23.063550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016f118f-689d-446e-bde1-a96e4c6c3e2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:23.066676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016f118f-689d-446e-bde1-a96e4c6c3e2e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:23.091201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1332/2621] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:23.094248 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:23.115819 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5cd4ee-2be5-4243-8a1e-57a14fa9524c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:23.154899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:23.156656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:23.157133 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:23.160554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] PATH_INFO: `/v3/projects/7a9f8833ef7d44fa885778b1eb866fe4/users/8f9e0997f1af4477b928b0724deb999d/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:23.165506 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a9f8833ef7d44fa885778b1eb866fe4, user_id=8f9e0997f1af4477b928b0724deb999d, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:23.172990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:23.210464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1290/2622] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:23 2022] PUT /identity/v3/projects/7a9f8833ef7d44fa885778b1eb866fe4/users/8f9e0997f1af4477b928b0724deb999d/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:23.229945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9582776f-6447-4a23-85a5-a4ee6ac6c049 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:23.230642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9582776f-6447-4a23-85a5-a4ee6ac6c049 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:23.231262 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9582776f-6447-4a23-85a5-a4ee6ac6c049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:23.320941 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9582776f-6447-4a23-85a5-a4ee6ac6c049 None None] MFA Rules not processed for user `8f9e0997f1af4477b928b0724deb999d`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:23.322165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9582776f-6447-4a23-85a5-a4ee6ac6c049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:23.545478 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1333/2623] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:23 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:23.548006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:23.551539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:23.660997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85667a90-9636-4eec-807a-0fe1de433efb admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:23.665888 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:23.666811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:23.667178 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:23.667735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:23.681024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:23.682550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:23.702555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1291/2624] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:23 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:48:23.929683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9582776f-6447-4a23-85a5-a4ee6ac6c049 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:23.956445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9582776f-6447-4a23-85a5-a4ee6ac6c049 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:23.964711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7309606-9949-416e-ad01-c61f1ea31d0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:23.966142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7309606-9949-416e-ad01-c61f1ea31d0d service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:23.968312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7309606-9949-416e-ad01-c61f1ea31d0d service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:23.969494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7309606-9949-416e-ad01-c61f1ea31d0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:24.130331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7309606-9949-416e-ad01-c61f1ea31d0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:24.134016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7309606-9949-416e-ad01-c61f1ea31d0d service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:24.145535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:24.165381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1334/2625] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:23 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:24.836587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:24.838101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:24.844720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:24.947308 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a61b22-dbe8-43b6-b68b-5887be2e07bc service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:24.952698 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:24.954413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:24.954855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:24.955188 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] PATH_INFO: `/v3/users/e298631534ec44aaa2bda10a9a848b10` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:24.956831 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e298631534ec44aaa2bda10a9a848b10)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:24.959163 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:25.016771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e298631534ec44aaa2bda10a9a848b10'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:25.025026 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e298631534ec44aaa2bda10a9a848b10'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:25.035252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e298631534ec44aaa2bda10a9a848b10'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:25.035926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e298631534ec44aaa2bda10a9a848b10'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:25.053800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e298631534ec44aaa2bda10a9a848b10'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:25.061332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1292/2626] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:24 2022] DELETE /identity/v3/users/e298631534ec44aaa2bda10a9a848b10 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:25.066348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071453e7-fca3-437b-9360-7b4da4a27bd3 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:25.067074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071453e7-fca3-437b-9360-7b4da4a27bd3 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:25.067074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071453e7-fca3-437b-9360-7b4da4a27bd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:25.095293 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-071453e7-fca3-437b-9360-7b4da4a27bd3 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:25.096632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-071453e7-fca3-437b-9360-7b4da4a27bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:25.253776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1335/2627] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:25.278829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:25.282468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:25.517460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a2802-548b-4282-9176-ad82d15205e1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:25.526993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:25.527926 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:25.528080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:25.528454 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:25.542056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:25.543624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:25.557332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1293/2628] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:25.920598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071453e7-fca3-437b-9360-7b4da4a27bd3 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:25.939075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071453e7-fca3-437b-9360-7b4da4a27bd3 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:25.946265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1430612f-9d36-427b-ac8a-48ab217e1642 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:25.948154 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1430612f-9d36-427b-ac8a-48ab217e1642 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:25.948476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1430612f-9d36-427b-ac8a-48ab217e1642 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:25.950090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1430612f-9d36-427b-ac8a-48ab217e1642 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:25.954363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1430612f-9d36-427b-ac8a-48ab217e1642 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:25.986272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:25.987575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:26.040031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f804fc-24d8-4338-a7f5-c42a61fedc59 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:26.048464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:26.049553 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:26.050009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:26.050401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] PATH_INFO: `/v3/projects/75a17b8d19514c788f2e60fe728b4ca4` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:26.052990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] RBAC: Authorizing `identity:delete_project(project_id=75a17b8d19514c788f2e60fe728b4ca4)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:26.057202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:26.116328 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:26.165680 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1430612f-9d36-427b-ac8a-48ab217e1642 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:26.167388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1430612f-9d36-427b-ac8a-48ab217e1642 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:26.187343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1336/2629] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:25 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:26.191585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75a17b8d19514c788f2e60fe728b4ca4', 'request_id': 'req-b772b361-e961-4c71-b335-dbf8ae40657c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:26.200093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75a17b8d19514c788f2e60fe728b4ca4', 'request_id': 'req-b772b361-e961-4c71-b335-dbf8ae40657c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:26.212694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75a17b8d19514c788f2e60fe728b4ca4', 'request_id': 'req-b772b361-e961-4c71-b335-dbf8ae40657c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:26.245462 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] The token cache is being invalidate because project 75a17b8d19514c788f2e60fe728b4ca4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:26.245901 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:26.246660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b772b361-e961-4c71-b335-dbf8ae40657c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:26.249042 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1294/2630] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:25 2022] DELETE /identity/v3/projects/75a17b8d19514c788f2e60fe728b4ca4 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:26.254762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4febe0d-80f9-46df-b3fa-f5fd7abf8730 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:26.255436 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4febe0d-80f9-46df-b3fa-f5fd7abf8730 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:26.255953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4febe0d-80f9-46df-b3fa-f5fd7abf8730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:26.324410 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c4febe0d-80f9-46df-b3fa-f5fd7abf8730 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:26.327880 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c4febe0d-80f9-46df-b3fa-f5fd7abf8730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:26.510844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1337/2631] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:26.522301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a1cdc-a304-4adc-9520-3dd0ab96012a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:26.523431 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a1cdc-a304-4adc-9520-3dd0ab96012a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:26.526161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a1cdc-a304-4adc-9520-3dd0ab96012a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:26.570894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cf1a1cdc-a304-4adc-9520-3dd0ab96012a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:26.573600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cf1a1cdc-a304-4adc-9520-3dd0ab96012a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:26.589354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1295/2632] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:26.594871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4febe0d-80f9-46df-b3fa-f5fd7abf8730 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:26.619306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4febe0d-80f9-46df-b3fa-f5fd7abf8730 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:26.634110 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:26.637670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:26.638017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:26.638328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:26.640072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:26.640072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:26.645578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:26.681535 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1338/2633] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:48:26.691829 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a1cdc-a304-4adc-9520-3dd0ab96012a None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:26.721922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a1cdc-a304-4adc-9520-3dd0ab96012a None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:26.734455 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:26.736138 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:26.736731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:26.737331 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:26.738852 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:26.742873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:26.795153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1296/2634] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:26 2022] POST /identity/v3/projects => generated 371 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:26.796225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:26.808027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa07d15-e89c-4dae-b136-d553911cae9c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:26.838804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:26.922844 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:26.924781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:26.925555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:26.926050 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:26.927526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:26.931809 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:27.006033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1339/2635] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:26 2022] POST /identity/v3/users => generated 286 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:48:27.007978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:27.027538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47580e6-0ea4-4a74-9c41-9784aa1f9aec admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:27.038273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:27.039209 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.039598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.039999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.040868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:27.042763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:27.055024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1297/2636] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:27 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:27.059418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:27.072572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a865028-6d40-4437-a7df-0914d9ae91ff admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:27.080546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c018fa8c-356c-4ffd-bbbd-3098616605ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:27.081596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018fa8c-356c-4ffd-bbbd-3098616605ca admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.082477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018fa8c-356c-4ffd-bbbd-3098616605ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.083289 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018fa8c-356c-4ffd-bbbd-3098616605ca admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.084411 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c018fa8c-356c-4ffd-bbbd-3098616605ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:27.086464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c018fa8c-356c-4ffd-bbbd-3098616605ca admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:27.099681 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1340/2637] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:27 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:48:27.104705 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:27.124318 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed2e96b-f858-4f26-a963-ef422b4dce44 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:27.138767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:27.140385 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.141337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.141760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] PATH_INFO: `/v3/projects/23a54d0a3442474db1829cb42e2b3c95/users/6e9ce7b0fdbd43f28d38a4b5657a3e2e/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.164942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] RBAC: Authorizing `identity:create_grant(project_id=23a54d0a3442474db1829cb42e2b3c95, user_id=6e9ce7b0fdbd43f28d38a4b5657a3e2e, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:27.168712 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:27.189498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1298/2638] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:27 2022] PUT /identity/v3/projects/23a54d0a3442474db1829cb42e2b3c95/users/6e9ce7b0fdbd43f28d38a4b5657a3e2e/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:27.195872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.196373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.196460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.223912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:27.225484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:27.320059 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:27.322168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:27.322740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1341/2639] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:27.341231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c8aa7e-058c-4677-8344-6a8f93ee6d29 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:27.344724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:27.349496 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:27.350398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.350767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.350998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] PATH_INFO: `/v3/users/99a62e49e6d249c2a169ef7122e762ec` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.352522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=99a62e49e6d249c2a169ef7122e762ec)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:27.354561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:27.355683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:27.414990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99a62e49e6d249c2a169ef7122e762ec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.424240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99a62e49e6d249c2a169ef7122e762ec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.440874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99a62e49e6d249c2a169ef7122e762ec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.441806 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99a62e49e6d249c2a169ef7122e762ec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.465348 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ef695-fb0c-42a4-83ab-458f92c3a960 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:27.465874 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99a62e49e6d249c2a169ef7122e762ec'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.469867 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741f2f46-34e3-4296-b8a4-0c4a960a4b58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:27.471319 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741f2f46-34e3-4296-b8a4-0c4a960a4b58 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.471601 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741f2f46-34e3-4296-b8a4-0c4a960a4b58 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.471887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741f2f46-34e3-4296-b8a4-0c4a960a4b58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.474109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-741f2f46-34e3-4296-b8a4-0c4a960a4b58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:27.478546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1299/2640] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:27 2022] DELETE /identity/v3/users/99a62e49e6d249c2a169ef7122e762ec => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:27.494494 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741f2f46-34e3-4296-b8a4-0c4a960a4b58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:27.496217 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741f2f46-34e3-4296-b8a4-0c4a960a4b58 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:27.509757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1342/2641] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:27.715099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:27.717767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:27.757894 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=79,keystone:DELETE=17,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:27.829606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa36f2c-99f6-4bb1-84af-6d11081d63ac admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:27.835735 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:27.836768 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.837071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.837340 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] PATH_INFO: `/v3/projects/f6606b37accd43619058aef23f10745b` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.838978 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6606b37accd43619058aef23f10745b)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:27.841096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:27.911640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6606b37accd43619058aef23f10745b', 'request_id': 'req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.919824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6606b37accd43619058aef23f10745b', 'request_id': 'req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.934025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6606b37accd43619058aef23f10745b', 'request_id': 'req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.952437 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] The token cache is being invalidate because project f6606b37accd43619058aef23f10745b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:27.952966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.954079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-32086ef9-62a7-4e86-99e7-04f0fdc5e3da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:27.956841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1300/2642] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:27 2022] DELETE /identity/v3/projects/f6606b37accd43619058aef23f10745b => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:27.962868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5319f8-c7ad-4d4a-989e-6395d258292b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:27.963351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5319f8-c7ad-4d4a-989e-6395d258292b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:27.963789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5319f8-c7ad-4d4a-989e-6395d258292b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:27.989921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-fb5319f8-c7ad-4d4a-989e-6395d258292b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:27.993181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-fb5319f8-c7ad-4d4a-989e-6395d258292b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:28.057048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:28.094080 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1343/2643] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:28.101344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38520e7-c601-4ef0-8834-4984618b88e7 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.102210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38520e7-c601-4ef0-8834-4984618b88e7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.102210 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38520e7-c601-4ef0-8834-4984618b88e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.130498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-a38520e7-c601-4ef0-8834-4984618b88e7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:28.132046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a38520e7-c601-4ef0-8834-4984618b88e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:28.143355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1301/2644] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:28.149035 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5319f8-c7ad-4d4a-989e-6395d258292b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:28.165919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5319f8-c7ad-4d4a-989e-6395d258292b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:28.173090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:28.174664 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.175284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.175976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.177165 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:28.177768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:28.179929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:28.199038 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1344/2645] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:48:28.208589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38520e7-c601-4ef0-8834-4984618b88e7 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:28.219856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38520e7-c601-4ef0-8834-4984618b88e7 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:28.225693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:28.226670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.227160 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.227644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.228659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:28.230854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:28.300499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1302/2646] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:28 2022] POST /identity/v3/projects => generated 373 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:28.307555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:28.321731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f71773a-604a-4aa2-982d-33eacb43b2dd admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:28.405522 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:28.407695 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.408341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.408962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.410201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:28.413267 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:28.444316 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1345/2647] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:28 2022] POST /identity/v3/users => generated 287 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:48:28.451301 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:28.474752 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69982a1d-d29a-491f-b87b-2922cc97f3e4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:28.483005 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:28.484034 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.484353 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.484608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.485428 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:28.487215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:28.518405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1303/2648] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:28 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:28.525939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:28.544814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a554b42-b886-43fb-b02e-f4b7f3dfc6d5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:28.552942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e08ebc-64b4-436e-a96b-799fd67bc7fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:28.554096 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e08ebc-64b4-436e-a96b-799fd67bc7fe admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.554687 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e08ebc-64b4-436e-a96b-799fd67bc7fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.555193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e08ebc-64b4-436e-a96b-799fd67bc7fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.556658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e08ebc-64b4-436e-a96b-799fd67bc7fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:28.559907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e08ebc-64b4-436e-a96b-799fd67bc7fe admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:28.585762 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1346/2649] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:28 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:28.586918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:28.600892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5ac654-10e4-4c13-8834-bff7636ee428 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:28.606230 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:28.607626 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.607912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.608190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] PATH_INFO: `/v3/projects/8bb9c3c7a1e54ced9dd9a1d40e736cb4/users/e4c1e471f9604b269ea04ac173504e7f/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.632640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bb9c3c7a1e54ced9dd9a1d40e736cb4, user_id=e4c1e471f9604b269ea04ac173504e7f, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:28.635729 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:28.657703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1304/2650] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:28 2022] PUT /identity/v3/projects/8bb9c3c7a1e54ced9dd9a1d40e736cb4/users/e4c1e471f9604b269ea04ac173504e7f/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:28.668476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5961d959-2e8f-4ff4-83f4-655be111f249 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.668799 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5961d959-2e8f-4ff4-83f4-655be111f249 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.669051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5961d959-2e8f-4ff4-83f4-655be111f249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.709734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-5961d959-2e8f-4ff4-83f4-655be111f249 None None] MFA Rules not processed for user `e4c1e471f9604b269ea04ac173504e7f`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:28.711188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5961d959-2e8f-4ff4-83f4-655be111f249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:28.804373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1347/2651] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:28 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:28.818656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:28.820028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:28.893396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:28.974314 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b1d442-e290-4dfb-896a-175bc3941c37 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:28.975043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b1d442-e290-4dfb-896a-175bc3941c37 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:28.975391 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b1d442-e290-4dfb-896a-175bc3941c37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:28.987265 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fbbd89-8a6f-42d6-9e55-f1cbebec1aca admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:28.998596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:29.001421 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:29.002124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:29.002660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:29.027231 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:29.032790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:29.049043 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-52b1d442-e290-4dfb-896a-175bc3941c37 None None] MFA Rules not processed for user `6e9ce7b0fdbd43f28d38a4b5657a3e2e`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:29.051720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-52b1d442-e290-4dfb-896a-175bc3941c37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:29.058878 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1305/2652] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:28 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:48:29.105227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:29.179320 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1348/2653] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:28 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:29.197619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:29.200214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:29.408637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dd7d72-e1ca-4e09-b5d4-a1f00c308e5d service nova] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:29.424444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:29.425526 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:29.426009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:29.426343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:29.449459 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:29.458050 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:29.489277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1306/2654] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:29 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:31.262142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810bf3fc-29c2-490a-8318-e0b7b4d8ec6f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:31.262815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810bf3fc-29c2-490a-8318-e0b7b4d8ec6f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:31.262815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810bf3fc-29c2-490a-8318-e0b7b4d8ec6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:31.290199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-810bf3fc-29c2-490a-8318-e0b7b4d8ec6f None None] MFA Rules not processed for user `6e9ce7b0fdbd43f28d38a4b5657a3e2e`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:31.292160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-810bf3fc-29c2-490a-8318-e0b7b4d8ec6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:31.303487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1349/2655] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:32.304558 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:32.396256 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539a82d2-ce4d-4ca1-8705-dabaf08b2ba1 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:32.404445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:32.405656 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:32.407153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:32.407590 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] PATH_INFO: `/v3/users/e4c1e471f9604b269ea04ac173504e7f` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:32.409241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4c1e471f9604b269ea04ac173504e7f)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:32.412711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:32.478375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:32.481511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4c1e471f9604b269ea04ac173504e7f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.492699 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4c1e471f9604b269ea04ac173504e7f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.503037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4c1e471f9604b269ea04ac173504e7f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.503688 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4c1e471f9604b269ea04ac173504e7f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.519324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4c1e471f9604b269ea04ac173504e7f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.527675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1307/2656] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:32 2022] DELETE /identity/v3/users/e4c1e471f9604b269ea04ac173504e7f => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:32.534034 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:32.534791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:32.534791 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:32.555057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:32.556472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:32.646256 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1350/2657] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:32.664096 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:32.666511 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:32.800629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5b4304-77dd-41b7-892f-3b089fd74aff admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:32.807100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f355942d-b59d-4767-b7fc-c766f6ebf206 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:32.808190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f355942d-b59d-4767-b7fc-c766f6ebf206 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:32.809723 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f355942d-b59d-4767-b7fc-c766f6ebf206 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:32.810286 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f355942d-b59d-4767-b7fc-c766f6ebf206 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:32.817623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:32.819371 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:32.826016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f355942d-b59d-4767-b7fc-c766f6ebf206 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:32.827710 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f355942d-b59d-4767-b7fc-c766f6ebf206 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:32.836797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16612c-6cfe-47dc-9f73-485ad8946d33 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:32.839730 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1308/2658] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:32.843361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:32.844209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:32.844498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:32.844776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] PATH_INFO: `/v3/users/876de5a9796147318d858794d43dae1a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:32.846224 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] RBAC: Authorizing `identity:delete_user(user_id=876de5a9796147318d858794d43dae1a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:32.848346 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:32.944341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '876de5a9796147318d858794d43dae1a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.960830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '876de5a9796147318d858794d43dae1a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.961676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '876de5a9796147318d858794d43dae1a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.971855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '876de5a9796147318d858794d43dae1a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:32.993023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '876de5a9796147318d858794d43dae1a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:33.006787 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1351/2659] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:32 2022] DELETE /identity/v3/users/876de5a9796147318d858794d43dae1a => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:33.008649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.009097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.009337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.030551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:33.032580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:33.125561 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1309/2660] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:33.136127 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:33.137827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:33.188565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:33.217672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2362192e-d416-428d-8df7-dac3736f9f22 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:33.222389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88a8adf-09d0-4e8c-bfbf-c15fa3c6eb9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:33.223283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88a8adf-09d0-4e8c-bfbf-c15fa3c6eb9a service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.223579 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88a8adf-09d0-4e8c-bfbf-c15fa3c6eb9a service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.223943 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88a8adf-09d0-4e8c-bfbf-c15fa3c6eb9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.227470 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:33.229242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:33.236623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88a8adf-09d0-4e8c-bfbf-c15fa3c6eb9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:33.238199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88a8adf-09d0-4e8c-bfbf-c15fa3c6eb9a service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:33.251100 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9318bb2-9003-498d-ae58-0cd6f6e13727 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:33.252118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1352/2661] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:33.263460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:33.265124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.267040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.267689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] PATH_INFO: `/v3/projects/8bb9c3c7a1e54ced9dd9a1d40e736cb4` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.272357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bb9c3c7a1e54ced9dd9a1d40e736cb4)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:33.274889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:33.339499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:33.356743 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bb9c3c7a1e54ced9dd9a1d40e736cb4', 'request_id': 'req-c0c11be9-c962-43a3-99a8-c13f37f0b801'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:33.367144 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bb9c3c7a1e54ced9dd9a1d40e736cb4', 'request_id': 'req-c0c11be9-c962-43a3-99a8-c13f37f0b801'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:33.379070 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bb9c3c7a1e54ced9dd9a1d40e736cb4', 'request_id': 'req-c0c11be9-c962-43a3-99a8-c13f37f0b801'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:33.394401 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] The token cache is being invalidate because project 8bb9c3c7a1e54ced9dd9a1d40e736cb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:33.394866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:33.395800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-c0c11be9-c962-43a3-99a8-c13f37f0b801 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:33.397472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1310/2662] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:33 2022] DELETE /identity/v3/projects/8bb9c3c7a1e54ced9dd9a1d40e736cb4 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:33.404993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c24fb9-c431-498e-a85f-b18957e17189 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.405674 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c24fb9-c431-498e-a85f-b18957e17189 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.406859 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c24fb9-c431-498e-a85f-b18957e17189 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.445430 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e1c24fb9-c431-498e-a85f-b18957e17189 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:33.447085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e1c24fb9-c431-498e-a85f-b18957e17189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:33.551957 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1353/2663] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:33.554162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a3d1af-0e04-4caf-872c-194d0640032c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.554627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a3d1af-0e04-4caf-872c-194d0640032c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.554955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a3d1af-0e04-4caf-872c-194d0640032c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.588892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-39a3d1af-0e04-4caf-872c-194d0640032c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:33.591196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-39a3d1af-0e04-4caf-872c-194d0640032c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:33.618726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1311/2664] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:33.631171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c24fb9-c431-498e-a85f-b18957e17189 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:33.678453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c24fb9-c431-498e-a85f-b18957e17189 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:33.685508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:33.686793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.687360 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.687903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.689865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:33.690330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:33.692412 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:33.709726 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1354/2665] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:48:33.713600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a3d1af-0e04-4caf-872c-194d0640032c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:33.724530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a3d1af-0e04-4caf-872c-194d0640032c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:33.732973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:33.734208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.734776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.735269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.736565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:33.740371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:33.795557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1312/2666] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:33 2022] POST /identity/v3/projects => generated 367 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:33.797358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:33.810877 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba01836f-eff3-4b27-b3da-8f38986f0015 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:33.913008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:33.913874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.914244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.914540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.915265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:33.920555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:33.921987 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:33.925491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:33.950056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae26e39-ec7c-4ebf-8a16-e3a8212106e7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:33.958271 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:33.960173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.960479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.960895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] PATH_INFO: `/v3/projects/88e75c3700e14be18b49f2aff667d137` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:33.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1355/2667] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:33 2022] POST /identity/v3/users => generated 284 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:33.963556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=88e75c3700e14be18b49f2aff667d137)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:33.965793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:33.969333 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:33.991905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c27ad45-ce6c-4b3c-94d5-39401ba7db1b admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:33.997429 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:33.998563 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:33.998870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:33.999163 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:34.000532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:34.003105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:34.018869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1356/2668] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:33 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:48:34.022662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:34.040278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eada3a5-fc80-4d4c-b2f7-4fefa9204b7e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:34.058807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88e75c3700e14be18b49f2aff667d137', 'request_id': 'req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.071041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88e75c3700e14be18b49f2aff667d137', 'request_id': 'req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.085845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88e75c3700e14be18b49f2aff667d137', 'request_id': 'req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.107456 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] The token cache is being invalidate because project 88e75c3700e14be18b49f2aff667d137 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:34.108046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.109260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.112574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1313/2669] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:33 2022] DELETE /identity/v3/projects/88e75c3700e14be18b49f2aff667d137 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:34.118660 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:34.120436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:34.121389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:34.158211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:34.158856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:34.159155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:34.159422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:34.160175 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:34.161881 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:34.174086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1357/2670] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:34 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:34.178148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:34.180269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:34.197793 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f3a92-8ddb-4c7a-808f-88fe2a9f7dcf admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:34.205485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec6e7ca-8fe4-4b91-8b37-a060ec784cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:34.206735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6e7ca-8fe4-4b91-8b37-a060ec784cdc admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:34.207208 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6e7ca-8fe4-4b91-8b37-a060ec784cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:34.207636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6e7ca-8fe4-4b91-8b37-a060ec784cdc admin admin] PATH_INFO: `/v3/projects/3c83d78715ff4627b91f52751290691d/users/312422fc48be4d9cb66f410aa0fdef41/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:34.216627 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec08aa7e-d2fd-465d-b422-cd84dc1890cb admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:34.224601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:34.226298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:34.226830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:34.227375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] PATH_INFO: `/v3/users/8f9e0997f1af4477b928b0724deb999d` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:34.229591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f9e0997f1af4477b928b0724deb999d)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:34.232547 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:34.236621 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec6e7ca-8fe4-4b91-8b37-a060ec784cdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c83d78715ff4627b91f52751290691d, user_id=312422fc48be4d9cb66f410aa0fdef41, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:34.242195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec6e7ca-8fe4-4b91-8b37-a060ec784cdc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:34.264865 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1358/2671] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:34 2022] PUT /identity/v3/projects/3c83d78715ff4627b91f52751290691d/users/312422fc48be4d9cb66f410aa0fdef41/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:34.276623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=82,keystone:DELETE=15,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:34.278840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:34.279116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:34.279394 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:34.314159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] MFA Rules not processed for user `312422fc48be4d9cb66f410aa0fdef41`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:34.315623 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:34.317952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f9e0997f1af4477b928b0724deb999d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.331824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f9e0997f1af4477b928b0724deb999d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.345285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f9e0997f1af4477b928b0724deb999d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.346999 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f9e0997f1af4477b928b0724deb999d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.375666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f9e0997f1af4477b928b0724deb999d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:34.387767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1314/2672] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:34 2022] DELETE /identity/v3/users/8f9e0997f1af4477b928b0724deb999d => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:34.406713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1359/2673] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:34 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:34.418346 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:34.421479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:34.591670 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6504a667-9ee0-469d-bc4b-94283e7b634d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:34.596411 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6228c54e-03e8-4cfc-a79a-380ffe7cf53f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:34.597214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6228c54e-03e8-4cfc-a79a-380ffe7cf53f service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:34.597562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6228c54e-03e8-4cfc-a79a-380ffe7cf53f service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:34.597904 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6228c54e-03e8-4cfc-a79a-380ffe7cf53f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:34.692405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6228c54e-03e8-4cfc-a79a-380ffe7cf53f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:34.695240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6228c54e-03e8-4cfc-a79a-380ffe7cf53f service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:34.728357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1315/2674] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:34 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:34.862771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:34.867649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:34.974704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae98b420-9d54-4e2b-b095-9ee591bd1a5b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:35.081886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:35.115279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:35.116463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.116768 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.117054 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] PATH_INFO: `/v3/projects/7a9f8833ef7d44fa885778b1eb866fe4` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.119336 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a9f8833ef7d44fa885778b1eb866fe4)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:35.121926 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:35.190172 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a9f8833ef7d44fa885778b1eb866fe4', 'request_id': 'req-9006122a-b4f3-4fee-b535-7a2a72721045'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:35.203055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a9f8833ef7d44fa885778b1eb866fe4', 'request_id': 'req-9006122a-b4f3-4fee-b535-7a2a72721045'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:35.232400 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a9f8833ef7d44fa885778b1eb866fe4', 'request_id': 'req-9006122a-b4f3-4fee-b535-7a2a72721045'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:35.257018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] The token cache is being invalidate because project 7a9f8833ef7d44fa885778b1eb866fe4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:35.257362 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:35.258135 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-9006122a-b4f3-4fee-b535-7a2a72721045 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:35.259908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1360/2675] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:34 2022] DELETE /identity/v3/projects/7a9f8833ef7d44fa885778b1eb866fe4 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:35.267827 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847850b0-f44b-4d37-b610-7de5c5853e60 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.268435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847850b0-f44b-4d37-b610-7de5c5853e60 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.269008 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847850b0-f44b-4d37-b610-7de5c5853e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.299657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-847850b0-f44b-4d37-b610-7de5c5853e60 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:35.302164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-847850b0-f44b-4d37-b610-7de5c5853e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:35.362393 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:35.404184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1316/2676] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:35.413335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f3361-3d68-47b0-96bb-d88797a108c8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.413650 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f3361-3d68-47b0-96bb-d88797a108c8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.413912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f3361-3d68-47b0-96bb-d88797a108c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.448184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-8a8f3361-3d68-47b0-96bb-d88797a108c8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:35.449467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8a8f3361-3d68-47b0-96bb-d88797a108c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:35.466465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1361/2677] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:35.473580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847850b0-f44b-4d37-b610-7de5c5853e60 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:35.488291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847850b0-f44b-4d37-b610-7de5c5853e60 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:35.496357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:35.497690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.498499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.499127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.501432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:35.501925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:35.506179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:35.535453 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1317/2678] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:48:35.537435 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f3361-3d68-47b0-96bb-d88797a108c8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:35.552992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f3361-3d68-47b0-96bb-d88797a108c8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:35.559465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:35.560443 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.560869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.561285 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.563645 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:35.565983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:35.604202 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1362/2679] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:35 2022] POST /identity/v3/projects => generated 377 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:35.609202 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:35.626803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c30e8-7f1a-4593-ae47-eee38c59f78c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:35.713818 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:35.714720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.715130 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.715529 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.716383 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:35.718896 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:35.751693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1318/2680] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:35 2022] POST /identity/v3/users => generated 289 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:35.753932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:35.809263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46db712c-a470-46b8-9daf-b43f1a518aa5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:35.816465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:35.817491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.818307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.818965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.820984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:35.827063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:35.843313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1363/2681] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:35 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:35.849240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:35.864578 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052ff4c2-be06-4824-a65b-3d10d17d9eb5 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:35.871132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b14e497-4575-4779-8832-b737878af5f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:35.872217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14e497-4575-4779-8832-b737878af5f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.872728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14e497-4575-4779-8832-b737878af5f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.873305 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14e497-4575-4779-8832-b737878af5f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.874126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b14e497-4575-4779-8832-b737878af5f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:35.876235 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b14e497-4575-4779-8832-b737878af5f4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:35.893575 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1319/2682] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:35 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:35.899774 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:35.911725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d987b-68d1-4608-80ff-023929952077 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:35.916906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:35.918092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.918382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.918701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] PATH_INFO: `/v3/projects/925a1ed3aa37419a8c6a002e7e06ffb0/users/3160f841ae3645f785ab139216fe39e5/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:35.937619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] RBAC: Authorizing `identity:create_grant(project_id=925a1ed3aa37419a8c6a002e7e06ffb0, user_id=3160f841ae3645f785ab139216fe39e5, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:35.940418 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:35.957041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1364/2683] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:35 2022] PUT /identity/v3/projects/925a1ed3aa37419a8c6a002e7e06ffb0/users/3160f841ae3645f785ab139216fe39e5/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:35.968996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:35.969291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:35.969577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:36.001907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] MFA Rules not processed for user `3160f841ae3645f785ab139216fe39e5`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:36.003216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:36.082460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1320/2684] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:35 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:36.095422 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:36.097914 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:36.181444 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e31f7-41dd-43dd-8460-ae3dcd632344 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:36.185659 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335fb4a3-59de-486c-a616-cfc3df15a8fb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fb6c2e2ddb3b4d488fa25fa8e258fec0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:36.186347 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335fb4a3-59de-486c-a616-cfc3df15a8fb service glance] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:36.186661 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335fb4a3-59de-486c-a616-cfc3df15a8fb service glance] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:36.186962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335fb4a3-59de-486c-a616-cfc3df15a8fb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:36.197019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335fb4a3-59de-486c-a616-cfc3df15a8fb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:36.198565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335fb4a3-59de-486c-a616-cfc3df15a8fb service glance] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:36.209670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1365/2685] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:36 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:44.118374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:44.120831 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:44.191647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:44.233432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06686ae2-fdfd-4dd4-952e-6f2da9b4ca0b None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:44.241343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:44.243056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:44.243582 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:44.244095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] PATH_INFO: `/v3/users/312422fc48be4d9cb66f410aa0fdef41` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:44.246675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] RBAC: Authorizing `identity:delete_user(user_id=312422fc48be4d9cb66f410aa0fdef41)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:44.250585 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:44.345045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '312422fc48be4d9cb66f410aa0fdef41'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:44.374968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '312422fc48be4d9cb66f410aa0fdef41'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:44.374968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '312422fc48be4d9cb66f410aa0fdef41'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:44.375854 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '312422fc48be4d9cb66f410aa0fdef41'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:44.402543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '312422fc48be4d9cb66f410aa0fdef41'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:44.414503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1321/2686] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:44 2022] DELETE /identity/v3/users/312422fc48be4d9cb66f410aa0fdef41 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:44.420986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:44.421775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:44.421775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:44.442651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:44.443932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:44.471701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:44.529484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1366/2687] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:44.541497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:44.543920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:44.639571 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a652bd8a-e229-490d-93c3-cff12882f565 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:44.644851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5c0c2b-593c-4b36-8149-dac4bdbc271a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:44.645807 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c0c2b-593c-4b36-8149-dac4bdbc271a service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:44.646206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c0c2b-593c-4b36-8149-dac4bdbc271a service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:44.646628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c0c2b-593c-4b36-8149-dac4bdbc271a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:44.667573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5c0c2b-593c-4b36-8149-dac4bdbc271a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:44.669311 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5c0c2b-593c-4b36-8149-dac4bdbc271a service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:44.687871 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1322/2688] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:45.154730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.156631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:45.172139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a215f0a7-056a-45b5-89ef-318a546c0be2 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:45.178875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:45.180493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.180493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.180779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] PATH_INFO: `/v3/projects/3c83d78715ff4627b91f52751290691d` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.182278 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c83d78715ff4627b91f52751290691d)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:45.184374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:45.253104 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c83d78715ff4627b91f52751290691d', 'request_id': 'req-1294b0d7-1223-488f-9957-cb90315e8cb4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:45.263961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c83d78715ff4627b91f52751290691d', 'request_id': 'req-1294b0d7-1223-488f-9957-cb90315e8cb4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:45.272637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c83d78715ff4627b91f52751290691d', 'request_id': 'req-1294b0d7-1223-488f-9957-cb90315e8cb4'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:45.285654 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] The token cache is being invalidate because project 3c83d78715ff4627b91f52751290691d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:45.285948 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:45.286724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1294b0d7-1223-488f-9957-cb90315e8cb4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:45.288405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1367/2689] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:45 2022] DELETE /identity/v3/projects/3c83d78715ff4627b91f52751290691d => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:45.293941 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94901bc3-c349-499b-9f6d-8bb4d1ef7c7c None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.294601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94901bc3-c349-499b-9f6d-8bb4d1ef7c7c None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.294601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94901bc3-c349-499b-9f6d-8bb4d1ef7c7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.314825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-94901bc3-c349-499b-9f6d-8bb4d1ef7c7c None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:45.316079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-94901bc3-c349-499b-9f6d-8bb4d1ef7c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:45.373943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:45.389532 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1323/2690] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:45.399490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b2f323-27d3-44ad-9b1c-a8b6f9e6010b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.399781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b2f323-27d3-44ad-9b1c-a8b6f9e6010b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.400061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b2f323-27d3-44ad-9b1c-a8b6f9e6010b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.419876 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b4b2f323-27d3-44ad-9b1c-a8b6f9e6010b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:45.421254 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b4b2f323-27d3-44ad-9b1c-a8b6f9e6010b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:45.429916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1368/2691] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:45.434989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94901bc3-c349-499b-9f6d-8bb4d1ef7c7c None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.453908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94901bc3-c349-499b-9f6d-8bb4d1ef7c7c None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:45.463014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:45.464247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.464767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.465282 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.466923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:45.467507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:45.470591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:45.489731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1324/2692] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:48:45.493590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b2f323-27d3-44ad-9b1c-a8b6f9e6010b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.511896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b2f323-27d3-44ad-9b1c-a8b6f9e6010b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:45.520599 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:45.521754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.522277 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.522817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.524174 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:45.527986 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:45.564718 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1369/2693] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:45 2022] POST /identity/v3/projects => generated 403 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:48:45.569406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.582217 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fcca3d-774d-483a-8d6a-3452ed35176d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:45.652931 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:45.653534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.653824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.654106 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.654883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:45.657419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:45.680080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1325/2694] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:45 2022] POST /identity/v3/users => generated 302 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:48:45.684446 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.695829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773905b-aedc-46b6-b6a7-90088a1bbd72 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:45.701457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:45.702786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.703086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.703366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.704156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:45.705803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:45.717998 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1370/2695] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:45.723234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.735238 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd092ec-9fbb-48be-a4f5-2b6c906ffaf2 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:45.740253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69efe76-878d-42b8-8e05-817da0eca8b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:45.740946 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69efe76-878d-42b8-8e05-817da0eca8b0 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.741234 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69efe76-878d-42b8-8e05-817da0eca8b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.741518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69efe76-878d-42b8-8e05-817da0eca8b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.742274 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69efe76-878d-42b8-8e05-817da0eca8b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:45.743989 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69efe76-878d-42b8-8e05-817da0eca8b0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:45.755201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1326/2696] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:45.760150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.771905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a151c18f-60a1-42a3-8f47-d4925cf27e3a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:45.779250 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:45.780245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.780707 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.781123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] PATH_INFO: `/v3/projects/af3da08549e24aa09f9031fbb27251a7/users/902bbab4fff648e7a1db5d49afa6d323/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.799779 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] RBAC: Authorizing `identity:create_grant(project_id=af3da08549e24aa09f9031fbb27251a7, user_id=902bbab4fff648e7a1db5d49afa6d323, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:45.802829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:45.819244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1371/2697] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:45 2022] PUT /identity/v3/projects/af3da08549e24aa09f9031fbb27251a7/users/902bbab4fff648e7a1db5d49afa6d323/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:45.825742 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cbaf27-b1c2-4a51-8bdc-2c34684faf7f None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:45.826260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cbaf27-b1c2-4a51-8bdc-2c34684faf7f None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:45.826799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cbaf27-b1c2-4a51-8bdc-2c34684faf7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:45.850080 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-70cbaf27-b1c2-4a51-8bdc-2c34684faf7f None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:45.851417 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-70cbaf27-b1c2-4a51-8bdc-2c34684faf7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:45.922849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1327/2698] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:45.933321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:45.935116 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:45.987548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:46.003343 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88c4588-32fa-4308-b45b-d3ee17b6e72d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:46.008249 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:46.009107 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:46.009507 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:46.009899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:46.020731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:46.022248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:46.034961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1372/2699] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:48:47.426479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f60219-5183-4dc8-846c-b20b3b5f82e1 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:47.427296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f60219-5183-4dc8-846c-b20b3b5f82e1 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:47.427296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f60219-5183-4dc8-846c-b20b3b5f82e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:47.505392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-02f60219-5183-4dc8-846c-b20b3b5f82e1 None None] MFA Rules not processed for user `902bbab4fff648e7a1db5d49afa6d323`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:47.507683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-02f60219-5183-4dc8-846c-b20b3b5f82e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:47.694653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:47.754668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1328/2700] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:47 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:47.758866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:47.780324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7c2447-e714-47c2-9070-9072ce0a8a71 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:47.791669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:47.793196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:47.793558 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:47.794487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:47.799554 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:47.803740 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:47.857209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1373/2701] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:47 2022] POST /identity/v3/projects => generated 405 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:47.864498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f60219-5183-4dc8-846c-b20b3b5f82e1 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:47.885770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f60219-5183-4dc8-846c-b20b3b5f82e1 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:47.975119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:47.975822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:47.976157 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:47.976478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:47.977323 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:47.979544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:48.003046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1329/2702] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:47 2022] POST /identity/v3/users => generated 309 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:48.008764 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:48.020789 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715a6454-c2b4-49cd-8f4f-3433e8cff760 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:48.025860 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:48.026588 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:48.026878 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:48.027159 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:48.027923 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:48.029598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:48.042266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1374/2703] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:48.047079 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:48.059113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148343a1-e3f5-44ee-9c0c-9e7550c06be1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:48.067336 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:48.068856 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:48.069339 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:48.069799 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] PATH_INFO: `/v3/projects/b323736c17494633bc8efd3a56b4b0e3/users/8648f44942a64a44a84be55991a37f4a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:48.099334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b323736c17494633bc8efd3a56b4b0e3, user_id=8648f44942a64a44a84be55991a37f4a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:48.104645 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:48.128129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1330/2704] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:48 2022] PUT /identity/v3/projects/b323736c17494633bc8efd3a56b4b0e3/users/8648f44942a64a44a84be55991a37f4a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:48.132533 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:48.147354 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a925e14b-2145-4d57-9da4-c912ac3ffc66 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:48.222383 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528f360d-4428-4e5a-ab1b-5700509d0ff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:48.223191 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f360d-4428-4e5a-ab1b-5700509d0ff7 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:48.223491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f360d-4428-4e5a-ab1b-5700509d0ff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:48.223772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f360d-4428-4e5a-ab1b-5700509d0ff7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:48.224532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528f360d-4428-4e5a-ab1b-5700509d0ff7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:48.226759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528f360d-4428-4e5a-ab1b-5700509d0ff7 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:48.248814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1375/2705] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:48:48.250774 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:48.269036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64f4a63-0f1f-4bf6-9236-03dfc7db9045 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:48.277129 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534d4dfd-c852-4956-b9ea-90f13ff90b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:48.278460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d4dfd-c852-4956-b9ea-90f13ff90b23 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:48.278770 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d4dfd-c852-4956-b9ea-90f13ff90b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:48.279046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d4dfd-c852-4956-b9ea-90f13ff90b23 admin admin] PATH_INFO: `/v3/projects/b323736c17494633bc8efd3a56b4b0e3/users/8648f44942a64a44a84be55991a37f4a/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:48.282270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534d4dfd-c852-4956-b9ea-90f13ff90b23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b323736c17494633bc8efd3a56b4b0e3, user_id=8648f44942a64a44a84be55991a37f4a, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:48.286227 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534d4dfd-c852-4956-b9ea-90f13ff90b23 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:48.305172 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1331/2706] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:48 2022] PUT /identity/v3/projects/b323736c17494633bc8efd3a56b4b0e3/users/8648f44942a64a44a84be55991a37f4a/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:48.875027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c697a069-d7fc-4ea6-8901-aedb2b2e4d61 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:48.875027 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c697a069-d7fc-4ea6-8901-aedb2b2e4d61 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:48.876222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c697a069-d7fc-4ea6-8901-aedb2b2e4d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:48.899178 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-c697a069-d7fc-4ea6-8901-aedb2b2e4d61 None None] MFA Rules not processed for user `6e9ce7b0fdbd43f28d38a4b5657a3e2e`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:48.900550 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c697a069-d7fc-4ea6-8901-aedb2b2e4d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:48.968548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1376/2707] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:49.147497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24b9b0-1037-4920-a201-28e852f00690 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:49.148392 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24b9b0-1037-4920-a201-28e852f00690 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:49.148610 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24b9b0-1037-4920-a201-28e852f00690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:49.190566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-7f24b9b0-1037-4920-a201-28e852f00690 None None] MFA Rules not processed for user `8648f44942a64a44a84be55991a37f4a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:49.192296 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7f24b9b0-1037-4920-a201-28e852f00690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:49.340867 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1332/2708] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:49 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:51.067118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c697a069-d7fc-4ea6-8901-aedb2b2e4d61 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:51.173392 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:51.207798 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c697a069-d7fc-4ea6-8901-aedb2b2e4d61 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:51.219093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:51.220624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:51.221302 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:51.221819 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] PATH_INFO: `/v3/users/902bbab4fff648e7a1db5d49afa6d323` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:51.224238 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=902bbab4fff648e7a1db5d49afa6d323)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:51.228290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:51.341338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '902bbab4fff648e7a1db5d49afa6d323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:51.372983 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '902bbab4fff648e7a1db5d49afa6d323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:51.375632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '902bbab4fff648e7a1db5d49afa6d323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:51.402665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '902bbab4fff648e7a1db5d49afa6d323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:51.440761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '902bbab4fff648e7a1db5d49afa6d323'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:51.458746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1377/2709] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:51 2022] DELETE /identity/v3/users/902bbab4fff648e7a1db5d49afa6d323 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:51.749568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f24b9b0-1037-4920-a201-28e852f00690 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:51.766720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f24b9b0-1037-4920-a201-28e852f00690 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:51.804216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:51.862971 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:51.864765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:51.865307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:51.865805 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] PATH_INFO: `/v3/projects/af3da08549e24aa09f9031fbb27251a7` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:51.868067 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=af3da08549e24aa09f9031fbb27251a7)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:51.871992 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:51.967607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af3da08549e24aa09f9031fbb27251a7', 'request_id': 'req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:51.979254 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af3da08549e24aa09f9031fbb27251a7', 'request_id': 'req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:51.992319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af3da08549e24aa09f9031fbb27251a7', 'request_id': 'req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.014151 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] The token cache is being invalidate because project af3da08549e24aa09f9031fbb27251a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:52.015218 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.016409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.019251 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1333/2710] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:51 2022] DELETE /identity/v3/projects/af3da08549e24aa09f9031fbb27251a7 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:52.024331 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:52.026903 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:52.045137 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a2a67b-21e5-4dce-8849-f7b84fd571b5 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:52.157442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:52.158966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:52.159370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:52.159747 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] PATH_INFO: `/v3/users/8648f44942a64a44a84be55991a37f4a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:52.161396 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] RBAC: Authorizing `identity:delete_user(user_id=8648f44942a64a44a84be55991a37f4a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:52.164081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:52.257671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8648f44942a64a44a84be55991a37f4a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.276308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8648f44942a64a44a84be55991a37f4a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.277918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8648f44942a64a44a84be55991a37f4a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.294301 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8648f44942a64a44a84be55991a37f4a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.311290 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:52.321048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-20c44e8b-6b74-411a-8ad8-cf9c732d6bff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8648f44942a64a44a84be55991a37f4a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.333657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1378/2711] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:52 2022] DELETE /identity/v3/users/8648f44942a64a44a84be55991a37f4a => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:52.631628 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:52.634640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:52.668881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdce53-5a1e-498d-bc19-c72ded7b8e4c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:52.770594 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:52.772164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:52.772761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:52.773343 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] PATH_INFO: `/v3/projects/b323736c17494633bc8efd3a56b4b0e3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:52.775615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b323736c17494633bc8efd3a56b4b0e3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:52.778276 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:52.844272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b323736c17494633bc8efd3a56b4b0e3', 'request_id': 'req-144dc880-d162-4772-9188-6a323b0bd960'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.855491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b323736c17494633bc8efd3a56b4b0e3', 'request_id': 'req-144dc880-d162-4772-9188-6a323b0bd960'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.859542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:52.867161 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b323736c17494633bc8efd3a56b4b0e3', 'request_id': 'req-144dc880-d162-4772-9188-6a323b0bd960'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.882983 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] The token cache is being invalidate because project b323736c17494633bc8efd3a56b4b0e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:52.883295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.884206 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-144dc880-d162-4772-9188-6a323b0bd960 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:52.891354 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1334/2712] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:52 2022] DELETE /identity/v3/projects/b323736c17494633bc8efd3a56b4b0e3 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:52.894118 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0658ec-b1d7-4c93-a01c-d184e3d8dcc8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:52.894631 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0658ec-b1d7-4c93-a01c-d184e3d8dcc8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:52.895155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0658ec-b1d7-4c93-a01c-d184e3d8dcc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:52.927092 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ae0658ec-b1d7-4c93-a01c-d184e3d8dcc8 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:52.928976 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ae0658ec-b1d7-4c93-a01c-d184e3d8dcc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:53.045692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1379/2713] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:53.055375 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f0ee44-478c-44c1-ae27-02dbd062107b None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.055885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f0ee44-478c-44c1-ae27-02dbd062107b None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.056386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f0ee44-478c-44c1-ae27-02dbd062107b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.089749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-95f0ee44-478c-44c1-ae27-02dbd062107b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:53.092246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-95f0ee44-478c-44c1-ae27-02dbd062107b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:53.106588 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1335/2714] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:53.113220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0658ec-b1d7-4c93-a01c-d184e3d8dcc8 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.126565 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0658ec-b1d7-4c93-a01c-d184e3d8dcc8 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:53.132380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:53.133081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.133397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.133853 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.134994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:53.135492 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:53.137770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:53.153298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1380/2715] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:48:53.164658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f0ee44-478c-44c1-ae27-02dbd062107b None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.191484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f0ee44-478c-44c1-ae27-02dbd062107b None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:53.201899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:53.203329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.203892 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.204396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.205862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:53.210794 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:53.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1336/2716] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:53 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:53.279008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.296845 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f50b95-04e4-423f-bf1f-8340d9347e98 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:53.374061 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:53.375324 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.375829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.376527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.377934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:53.381873 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:53.413207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1381/2717] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:53 2022] POST /identity/v3/users => generated 293 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:53.419704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.438477 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5631e131-6b98-43c6-be2e-d8dc3664e331 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:53.447368 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:53.448786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.449377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.450001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.451649 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:53.455273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:53.476105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1337/2718] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:53.480439 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.492397 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be08e64-bd67-479e-9b41-029b306f9f10 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:53.497627 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f6287f-bbdf-43b9-927b-304783b21edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:53.498463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f6287f-bbdf-43b9-927b-304783b21edd admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.498863 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f6287f-bbdf-43b9-927b-304783b21edd admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.499210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f6287f-bbdf-43b9-927b-304783b21edd admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.500056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f6287f-bbdf-43b9-927b-304783b21edd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:53.501725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f6287f-bbdf-43b9-927b-304783b21edd admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:53.513099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1382/2719] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:53.520374 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.540095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2112ab31-05b8-4672-b76a-ea88d8ef9de7 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:53.548499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:53.550166 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.550741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.551294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] PATH_INFO: `/v3/projects/2b782bcd03504d32aecec6df7a78b2ef/users/17bfe452bbc24be8b8223343f8c541d9/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.581260 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b782bcd03504d32aecec6df7a78b2ef, user_id=17bfe452bbc24be8b8223343f8c541d9, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:53.587287 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:53.615440 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1338/2720] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:53 2022] PUT /identity/v3/projects/2b782bcd03504d32aecec6df7a78b2ef/users/17bfe452bbc24be8b8223343f8c541d9/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:53.634827 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf778f-93dd-49eb-9729-4cb17de63cbb None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.635325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf778f-93dd-49eb-9729-4cb17de63cbb None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.635842 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf778f-93dd-49eb-9729-4cb17de63cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.680245 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-21bf778f-93dd-49eb-9729-4cb17de63cbb None None] MFA Rules not processed for user `17bfe452bbc24be8b8223343f8c541d9`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:53.682087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-21bf778f-93dd-49eb-9729-4cb17de63cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:53.745861 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:53.770017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1383/2721] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:53 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:53.775083 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.867486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f219e07d-a98d-4aaa-b364-dce56802cf09 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:53.873087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:53.873791 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:53.874727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:53.874986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:53.876053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:53.878061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:53.910288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1339/2722] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:53 2022] POST /identity/v3/projects => generated 387 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:53.916087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf778f-93dd-49eb-9729-4cb17de63cbb None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:53.936142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf778f-93dd-49eb-9729-4cb17de63cbb None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:54.040532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:54.041636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:54.042088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:54.042561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:54.043708 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:54.047079 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:54.083283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1384/2723] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:53 2022] POST /identity/v3/users => generated 300 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:54.084996 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:54.097283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88e90a9-729c-4c22-b00b-b3fb687112d4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:54.102650 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:54.103313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:54.103603 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:54.103893 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:54.104655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:54.106597 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:54.118584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1340/2724] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:54.125991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:54.136663 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36e0e7a-c461-4142-9b40-7b157608d08c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:54.143767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:54.145437 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:54.145915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:54.146539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] PATH_INFO: `/v3/projects/b644a4422f97417d92f8a5e53d8eb287/users/a5031c4508394b8d9deb450d2d012bd2/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:54.176273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b644a4422f97417d92f8a5e53d8eb287, user_id=a5031c4508394b8d9deb450d2d012bd2, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:54.180294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:54.198965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1385/2725] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:54 2022] PUT /identity/v3/projects/b644a4422f97417d92f8a5e53d8eb287/users/a5031c4508394b8d9deb450d2d012bd2/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:54.203429 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:54.215881 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918e651e-ee42-4bd3-8f76-12412052fbb9 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:54.299266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495d394f-195f-4fcb-99ea-4a0f64b11e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:54.300681 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495d394f-195f-4fcb-99ea-4a0f64b11e59 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:54.301425 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495d394f-195f-4fcb-99ea-4a0f64b11e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:54.302155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495d394f-195f-4fcb-99ea-4a0f64b11e59 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:54.303637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495d394f-195f-4fcb-99ea-4a0f64b11e59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:54.306911 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495d394f-195f-4fcb-99ea-4a0f64b11e59 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:54.309920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:54.329467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1341/2726] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:54 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:54.336049 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:54.347731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02350031-7aab-4292-8639-e56e11e08df1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:54.353000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:54.353919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:54.354200 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:54.354518 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] PATH_INFO: `/v3/projects/b644a4422f97417d92f8a5e53d8eb287/users/a5031c4508394b8d9deb450d2d012bd2/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:54.356741 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=b644a4422f97417d92f8a5e53d8eb287, user_id=a5031c4508394b8d9deb450d2d012bd2, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:54.360094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:54.377937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1386/2727] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:54 2022] PUT /identity/v3/projects/b644a4422f97417d92f8a5e53d8eb287/users/a5031c4508394b8d9deb450d2d012bd2/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:54.389798 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f0193-cc40-4003-a9c0-9a41c5c80bc0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:54.390089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f0193-cc40-4003-a9c0-9a41c5c80bc0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:54.390367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f0193-cc40-4003-a9c0-9a41c5c80bc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:54.420517 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-8a8f0193-cc40-4003-a9c0-9a41c5c80bc0 None None] MFA Rules not processed for user `a5031c4508394b8d9deb450d2d012bd2`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:54.422423 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8a8f0193-cc40-4003-a9c0-9a41c5c80bc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:54.506205 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1342/2728] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:54 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:54.517885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:54.519932 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:54.588182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cc55af-96f7-40e7-817d-fee6079fd6dc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:54.591945 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b3a61f-d659-400e-a5d2-2748444c0b83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:54.593300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b3a61f-d659-400e-a5d2-2748444c0b83 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:54.593590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b3a61f-d659-400e-a5d2-2748444c0b83 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:54.593869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b3a61f-d659-400e-a5d2-2748444c0b83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:54.604739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b3a61f-d659-400e-a5d2-2748444c0b83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:54.606199 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b3a61f-d659-400e-a5d2-2748444c0b83 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:54.618266 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1387/2729] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:54 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:55.497216 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f0193-cc40-4003-a9c0-9a41c5c80bc0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:55.638858 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f0193-cc40-4003-a9c0-9a41c5c80bc0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:55.643959 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:55.645167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:55.645677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:55.646162 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] PATH_INFO: `/v3/users/17bfe452bbc24be8b8223343f8c541d9` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:55.647707 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] RBAC: Authorizing `identity:delete_user(user_id=17bfe452bbc24be8b8223343f8c541d9)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:55.650317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:55.741811 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17bfe452bbc24be8b8223343f8c541d9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:55.750657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17bfe452bbc24be8b8223343f8c541d9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:55.761432 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17bfe452bbc24be8b8223343f8c541d9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:55.762250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17bfe452bbc24be8b8223343f8c541d9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:55.778740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17bfe452bbc24be8b8223343f8c541d9'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:55.782158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:55.791637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1343/2730] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:55 2022] DELETE /identity/v3/users/17bfe452bbc24be8b8223343f8c541d9 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:55.799128 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:55.799713 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:55.799884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:55.826120 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:55.828056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:55.895913 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:55.924596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1388/2731] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:55.937584 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:55.939367 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:56.008294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4161d63-f4ab-4c6a-b4c6-3fdab7e95282 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:56.012637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:56.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:56.013918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:56.014297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:56.025310 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:56.027086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:56.038824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1344/2732] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:56.368727 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:56.371800 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:56.390365 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9cb8b1-bb33-4556-af90-51d1e3f5ea67 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:56.396184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:56.397105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:56.397417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:56.397709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] PATH_INFO: `/v3/projects/2b782bcd03504d32aecec6df7a78b2ef` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:56.399295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b782bcd03504d32aecec6df7a78b2ef)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:56.401350 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:56.497462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b782bcd03504d32aecec6df7a78b2ef', 'request_id': 'req-f3be281f-8b1b-437c-a662-4819a37586be'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.509291 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b782bcd03504d32aecec6df7a78b2ef', 'request_id': 'req-f3be281f-8b1b-437c-a662-4819a37586be'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.520731 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b782bcd03504d32aecec6df7a78b2ef', 'request_id': 'req-f3be281f-8b1b-437c-a662-4819a37586be'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.538356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] The token cache is being invalidate because project 2b782bcd03504d32aecec6df7a78b2ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:56.538720 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.539609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.542307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1389/2733] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:56 2022] DELETE /identity/v3/projects/2b782bcd03504d32aecec6df7a78b2ef => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:56.548118 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:56.559355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bde470-ec95-4506-96d2-a76681cabf68 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:56.625790 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:56.626826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:56.627264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:56.627587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] PATH_INFO: `/v3/users/a5031c4508394b8d9deb450d2d012bd2` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:56.629174 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5031c4508394b8d9deb450d2d012bd2)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:56.631228 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:56.697167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5031c4508394b8d9deb450d2d012bd2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.706581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5031c4508394b8d9deb450d2d012bd2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.718279 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5031c4508394b8d9deb450d2d012bd2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.719074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5031c4508394b8d9deb450d2d012bd2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.735269 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-ff6f4827-0b94-4052-a651-1f20771c9ea0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5031c4508394b8d9deb450d2d012bd2'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:56.743017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1345/2734] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:56 2022] DELETE /identity/v3/users/a5031c4508394b8d9deb450d2d012bd2 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:56.770679 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:56.773629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:56.799113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3be281f-8b1b-437c-a662-4819a37586be admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:56.907952 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:56.910014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:56.910802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:56.911142 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] PATH_INFO: `/v3/users/3160f841ae3645f785ab139216fe39e5` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:56.914122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=3160f841ae3645f785ab139216fe39e5)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:56.918282 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:57.006321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3160f841ae3645f785ab139216fe39e5'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.021786 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3160f841ae3645f785ab139216fe39e5'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.023048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3160f841ae3645f785ab139216fe39e5'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.038123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3160f841ae3645f785ab139216fe39e5'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.047001 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:57.063802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3160f841ae3645f785ab139216fe39e5'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.075851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1390/2735] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:48:56 2022] DELETE /identity/v3/users/3160f841ae3645f785ab139216fe39e5 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:57.082602 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.083380 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.083870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.110367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:57.112871 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:57.118253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:57.120589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:57.157607 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:57.221243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b741825-f978-4235-acd1-b34c6081d3ed admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:57.226147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:57.226640 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1346/2736] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:57.227075 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.227358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.227638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] PATH_INFO: `/v3/projects/b644a4422f97417d92f8a5e53d8eb287` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.228953 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b644a4422f97417d92f8a5e53d8eb287)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:57.230966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:57.241304 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:57.243215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:57.296778 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b644a4422f97417d92f8a5e53d8eb287', 'request_id': 'req-1c559d78-04ac-4fd4-ac1f-012045c02f7d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.307736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b644a4422f97417d92f8a5e53d8eb287', 'request_id': 'req-1c559d78-04ac-4fd4-ac1f-012045c02f7d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.316781 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b644a4422f97417d92f8a5e53d8eb287', 'request_id': 'req-1c559d78-04ac-4fd4-ac1f-012045c02f7d'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.321014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b4f79-c280-4eb7-9195-e828cf58f1e7 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:57.333374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] The token cache is being invalidate because project b644a4422f97417d92f8a5e53d8eb287 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:57.333673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.334486 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-1c559d78-04ac-4fd4-ac1f-012045c02f7d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:57.336312 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1391/2737] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:57 2022] DELETE /identity/v3/projects/b644a4422f97417d92f8a5e53d8eb287 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:57.341729 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f89db-9f02-4166-b315-6feee2110012 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.342021 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f89db-9f02-4166-b315-6feee2110012 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.342310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f89db-9f02-4166-b315-6feee2110012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.364450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2c8f89db-9f02-4166-b315-6feee2110012 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:57.365775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2c8f89db-9f02-4166-b315-6feee2110012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:57.400081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:57.400765 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.401054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.401508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.403677 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:57.486356 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1392/2738] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:57.497538 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09305cc-b138-4c33-9d8e-48a97a0bc46b None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.498571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09305cc-b138-4c33-9d8e-48a97a0bc46b None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.499210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09305cc-b138-4c33-9d8e-48a97a0bc46b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.522140 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:57.524942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:57.535902 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e09305cc-b138-4c33-9d8e-48a97a0bc46b None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:57.539894 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e09305cc-b138-4c33-9d8e-48a97a0bc46b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:57.553727 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1347/2739] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:57.572067 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1393/2740] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:57.572514 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:57.593970 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2e3b7f-9a60-41ad-87a0-437f8f3ecd11 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:57.599972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:57.601168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.601559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.602293 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.603270 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:57.603701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:57.605731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:57.626692 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1348/2741] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:48:57.630690 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09305cc-b138-4c33-9d8e-48a97a0bc46b None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:57.643773 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09305cc-b138-4c33-9d8e-48a97a0bc46b None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:57.650964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:57.652460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.652770 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.653056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.654232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:57.657536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:57.667313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:57.725540 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1394/2742] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:57 2022] POST /identity/v3/projects => generated 371 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:48:57.731365 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:57.752499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba938c0-528c-420d-b631-24194e13b6d0 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:57.783657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=100 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:57.883408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:57.884240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.884654 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.885299 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.886089 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:57.888297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:57.926899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1349/2743] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:57 2022] POST /identity/v3/users => generated 286 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:48:57.931963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:57.953918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cb61d9-416e-4c72-b3f2-1710e84c43dc admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:57.963269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:57.965000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:57.965000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:57.965000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:57.966797 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:57.970915 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:57.994609 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1395/2744] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:48:57.999450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.019745 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b720d-eae7-4051-89aa-d50aa3981c3f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.027842 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.029021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.029522 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.030581 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.032028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.035107 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.058826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1350/2745] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:58.060484 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.075405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95536c6-1b66-4c5d-9760-e7f2f34a979f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.084705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c8da51-bb51-48a9-ba11-06ef4423d829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.086330 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c8da51-bb51-48a9-ba11-06ef4423d829 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.086884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c8da51-bb51-48a9-ba11-06ef4423d829 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.087404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c8da51-bb51-48a9-ba11-06ef4423d829 admin admin] PATH_INFO: `/v3/projects/0e8b934cdedf4654bb82f62880293b7f/users/e04c679752fb459f84ac794a646d241a/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.121057 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c8da51-bb51-48a9-ba11-06ef4423d829 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e8b934cdedf4654bb82f62880293b7f, user_id=e04c679752fb459f84ac794a646d241a, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.127065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c8da51-bb51-48a9-ba11-06ef4423d829 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.154510 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1396/2746] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:58 2022] PUT /identity/v3/projects/0e8b934cdedf4654bb82f62880293b7f/users/e04c679752fb459f84ac794a646d241a/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:48:58.160491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.161612 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.162150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.162717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.163639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:58.187824 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360706cc-3fa4-473a-bb22-0cf1c7301259 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.197517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:58.199856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:58.266986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.267943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.268408 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.268796 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] PATH_INFO: `/v3/projects/925a1ed3aa37419a8c6a002e7e06ffb0` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.270460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] RBAC: Authorizing `identity:delete_project(project_id=925a1ed3aa37419a8c6a002e7e06ffb0)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.273134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.312303 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1397/2747] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:58.331502 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.334076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:58.365760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '925a1ed3aa37419a8c6a002e7e06ffb0', 'request_id': 'req-64ab6c03-e163-4d23-a98a-ba54b53d9396'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:58.374753 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '925a1ed3aa37419a8c6a002e7e06ffb0', 'request_id': 'req-64ab6c03-e163-4d23-a98a-ba54b53d9396'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:58.387298 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '925a1ed3aa37419a8c6a002e7e06ffb0', 'request_id': 'req-64ab6c03-e163-4d23-a98a-ba54b53d9396'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:58.403472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] The token cache is being invalidate because project 925a1ed3aa37419a8c6a002e7e06ffb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:48:58.403781 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:58.404565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-64ab6c03-e163-4d23-a98a-ba54b53d9396 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:48:58.407313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1351/2748] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:48:58 2022] DELETE /identity/v3/projects/925a1ed3aa37419a8c6a002e7e06ffb0 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:58.413666 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd67442f-dd50-48c5-b6d2-1d937920ce9e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.413967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd67442f-dd50-48c5-b6d2-1d937920ce9e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.414245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd67442f-dd50-48c5-b6d2-1d937920ce9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.430519 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9b1127-c5dc-4cdb-b1d1-a9fd228b7b01 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.435663 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-cd67442f-dd50-48c5-b6d2-1d937920ce9e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:58.437055 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-cd67442f-dd50-48c5-b6d2-1d937920ce9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:58.465997 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:58.527220 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1352/2749] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:58.535416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.536438 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.536814 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.537155 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.540601 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2146959-3e6f-4118-a3b9-447bce60482d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.540939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2146959-3e6f-4118-a3b9-447bce60482d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.541405 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2146959-3e6f-4118-a3b9-447bce60482d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.544479 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:58.563239 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.564811 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.565268 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-f2146959-3e6f-4118-a3b9-447bce60482d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:58.566709 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-f2146959-3e6f-4118-a3b9-447bce60482d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:58.576657 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1353/2750] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:58.578041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1398/2751] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:48:58.582828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2146959-3e6f-4118-a3b9-447bce60482d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.595476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2146959-3e6f-4118-a3b9-447bce60482d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.602052 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.602966 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.603379 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.603763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.604719 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:48:58.605137 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.607073 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.624539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1354/2752] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:48:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:48:58.630566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.643103 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f03b30-a67f-456f-9031-7523d250e4ca service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.649265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.650105 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.650592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.650960 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.652086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.654613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.668636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:58.704219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1399/2753] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:58 2022] POST /identity/v3/projects => generated 385 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:48:58.710002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.722704 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82055e-c5c6-4472-85a5-0ac7fd222543 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.810215 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.813099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.814259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.814912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.816483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.820406 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.857497 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1355/2754] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:48:58 2022] POST /identity/v3/users => generated 293 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:48:58.861857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.883898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca7384a-b148-4692-a871-8dffb46b9e75 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.892139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.893113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.893572 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.894028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.895517 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.899753 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.916220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1400/2755] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:58.922364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.937862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb2d80b-9be1-4219-aa73-9d29e2c550dd admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.944464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de036c52-05ac-425b-ad83-a2497f638dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.945378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de036c52-05ac-425b-ad83-a2497f638dbe admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.945792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de036c52-05ac-425b-ad83-a2497f638dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.946247 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de036c52-05ac-425b-ad83-a2497f638dbe admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:58.947257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de036c52-05ac-425b-ad83-a2497f638dbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:58.949474 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de036c52-05ac-425b-ad83-a2497f638dbe admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:58.963956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1356/2756] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:48:58.968887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:58.979858 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da94fc-8aa8-45f7-8b13-ded788f67513 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:58.985039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:58.986004 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:58.986294 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:58.986622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] PATH_INFO: `/v3/projects/12d85165fa0f47239a665c86a92ea447/users/0d9714af42194c78a08921c63e7260ba/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:59.007937 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] RBAC: Authorizing `identity:create_grant(project_id=12d85165fa0f47239a665c86a92ea447, user_id=0d9714af42194c78a08921c63e7260ba, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:59.011273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:59.037749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1401/2757] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:48:58 2022] PUT /identity/v3/projects/12d85165fa0f47239a665c86a92ea447/users/0d9714af42194c78a08921c63e7260ba/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:48:59.046737 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6018566-4bfd-4bcd-b095-d3dd8cd1ac6a None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:59.047167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6018566-4bfd-4bcd-b095-d3dd8cd1ac6a None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:59.047624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6018566-4bfd-4bcd-b095-d3dd8cd1ac6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:59.085222 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c6018566-4bfd-4bcd-b095-d3dd8cd1ac6a None None] MFA Rules not processed for user `0d9714af42194c78a08921c63e7260ba`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:59.087917 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c6018566-4bfd-4bcd-b095-d3dd8cd1ac6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:59.214493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1357/2758] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:59 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:48:59.235059 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:59.399464 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5776930a-5119-4fdd-8e83-5190c9c87607 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:48:59.407106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:48:59.408308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:59.408837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:59.409340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:59.426662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:48:59.429307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:48:59.448248 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1402/2759] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:48:59 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:48:59.665471 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2835041-8172-4aed-94c6-11c501c7ff29 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:48:59.667000 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2835041-8172-4aed-94c6-11c501c7ff29 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:48:59.667667 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2835041-8172-4aed-94c6-11c501c7ff29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:48:59.727472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:48:59.737621 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c2835041-8172-4aed-94c6-11c501c7ff29 None None] MFA Rules not processed for user `e04c679752fb459f84ac794a646d241a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:48:59.741244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c2835041-8172-4aed-94c6-11c501c7ff29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:48:59.857315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1358/2760] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:48:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:48:59.859732 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:48:59.862000 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:00.044389 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0299e06-b29d-46ba-b64b-2baa313ab0f6 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:00.054555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:00.055763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:00.056276 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:00.056769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:00.058051 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:00.062549 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:00.121284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1403/2761] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:48:59 2022] POST /identity/v3/projects => generated 369 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:00.127361 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2835041-8172-4aed-94c6-11c501c7ff29 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:00.151104 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2835041-8172-4aed-94c6-11c501c7ff29 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:00.282064 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:00.283862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:00.284859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:00.285386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:00.286855 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:00.292749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:00.345548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:00.354545 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1359/2762] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:49:00 2022] POST /identity/v3/users => generated 291 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:49:00.359167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12e1dea-6a08-4431-8932-b67430ad61e9 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:00.364870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:00.365576 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:00.365868 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:00.366190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:00.367011 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:00.368728 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:00.382308 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1404/2763] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:49:00.388386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:00.408062 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61649402-e074-4d68-92ad-282d5f9ba300 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:00.418527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:00.420147 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:00.420736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:00.421267 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] PATH_INFO: `/v3/projects/dbd46346d1d24fbda86f76a3873a503e/users/fbfcc3bd9fba4c0aa7b9d9df3cae51e8/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:00.453164 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbd46346d1d24fbda86f76a3873a503e, user_id=fbfcc3bd9fba4c0aa7b9d9df3cae51e8, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:00.458895 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:00.484060 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1360/2764] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:00 2022] PUT /identity/v3/projects/dbd46346d1d24fbda86f76a3873a503e/users/fbfcc3bd9fba4c0aa7b9d9df3cae51e8/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:00.489785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:00.509207 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b50b6f0-d54e-4723-b533-4305f8ba76ed admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:00.568094 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:00.605150 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ff8dff-63e4-4e4d-af96-9cef0d6e07e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:00.605756 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ff8dff-63e4-4e4d-af96-9cef0d6e07e1 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:00.606044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ff8dff-63e4-4e4d-af96-9cef0d6e07e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:00.606328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ff8dff-63e4-4e4d-af96-9cef0d6e07e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:00.607141 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ff8dff-63e4-4e4d-af96-9cef0d6e07e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:00.609488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ff8dff-63e4-4e4d-af96-9cef0d6e07e1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:00.621395 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1405/2765] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:00.626098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:00.638115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc3e798-4549-418f-99bb-5180dbee2179 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:00.644015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:00.645045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:00.645391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:00.645786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] PATH_INFO: `/v3/projects/dbd46346d1d24fbda86f76a3873a503e/users/fbfcc3bd9fba4c0aa7b9d9df3cae51e8/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:00.648826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbd46346d1d24fbda86f76a3873a503e, user_id=fbfcc3bd9fba4c0aa7b9d9df3cae51e8, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:00.652653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:00.666653 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1361/2766] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:00 2022] PUT /identity/v3/projects/dbd46346d1d24fbda86f76a3873a503e/users/fbfcc3bd9fba4c0aa7b9d9df3cae51e8/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:01.488818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06466be6-a7c0-46f2-bcb0-25b65ee2d492 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:01.489671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06466be6-a7c0-46f2-bcb0-25b65ee2d492 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:01.490196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06466be6-a7c0-46f2-bcb0-25b65ee2d492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:01.535442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-06466be6-a7c0-46f2-bcb0-25b65ee2d492 None None] MFA Rules not processed for user `fbfcc3bd9fba4c0aa7b9d9df3cae51e8`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:01.537830 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-06466be6-a7c0-46f2-bcb0-25b65ee2d492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:01.670453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1406/2767] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:01 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:49:01.696211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:01.807443 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232fe190-31fb-451a-ae25-1617ee830b1a admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:01.814595 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb37c68-b993-4921-9af5-8b55e6854bc8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:01.815835 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb37c68-b993-4921-9af5-8b55e6854bc8 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:01.816315 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb37c68-b993-4921-9af5-8b55e6854bc8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:01.816905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb37c68-b993-4921-9af5-8b55e6854bc8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:01.828134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:01.903625 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb37c68-b993-4921-9af5-8b55e6854bc8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:01.905836 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb37c68-b993-4921-9af5-8b55e6854bc8 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:01.916728 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1362/2768] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:01 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:49:02.387284 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06466be6-a7c0-46f2-bcb0-25b65ee2d492 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:02.461750 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06466be6-a7c0-46f2-bcb0-25b65ee2d492 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:02.468088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:02.469014 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:02.469299 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:02.469586 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] PATH_INFO: `/v3/users/0d9714af42194c78a08921c63e7260ba` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:02.471028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d9714af42194c78a08921c63e7260ba)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:02.473338 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:02.544702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d9714af42194c78a08921c63e7260ba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:02.557247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d9714af42194c78a08921c63e7260ba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:02.558235 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d9714af42194c78a08921c63e7260ba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:02.560738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:02.574886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d9714af42194c78a08921c63e7260ba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:02.597033 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d9714af42194c78a08921c63e7260ba'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:02.603916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1407/2769] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:02 2022] DELETE /identity/v3/users/0d9714af42194c78a08921c63e7260ba => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:02.609019 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:02.609478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:02.609853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:02.630605 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:02.632845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:02.743958 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1363/2770] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:02.755452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:02.757160 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:02.824916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674a08a-0c4d-4518-aba2-e91e6e19c1d1 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:02.829866 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:02.830682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:02.831062 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:02.831379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:02.842892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:02.844700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:02.856171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1408/2771] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:49:03.257570 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:03.259942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:03.282503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da301b95-83be-4013-bf75-b0d4a6424d3d None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:03.291232 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:03.292889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:03.295186 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:03.299155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] PATH_INFO: `/v3/projects/12d85165fa0f47239a665c86a92ea447` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:03.303759 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] RBAC: Authorizing `identity:delete_project(project_id=12d85165fa0f47239a665c86a92ea447)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:03.307609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:03.376368 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:03.378994 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:03.410007 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12d85165fa0f47239a665c86a92ea447', 'request_id': 'req-079c3084-c2c7-4e94-a0eb-de28ac01046f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:03.418763 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12d85165fa0f47239a665c86a92ea447', 'request_id': 'req-079c3084-c2c7-4e94-a0eb-de28ac01046f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:03.431703 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12d85165fa0f47239a665c86a92ea447', 'request_id': 'req-079c3084-c2c7-4e94-a0eb-de28ac01046f'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:03.451066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] The token cache is being invalidate because project 12d85165fa0f47239a665c86a92ea447 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:03.451490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:03.452880 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-079c3084-c2c7-4e94-a0eb-de28ac01046f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:03.466473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1364/2772] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:03 2022] DELETE /identity/v3/projects/12d85165fa0f47239a665c86a92ea447 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:03.507082 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b614b559-8a6f-4c35-aef1-4f0dbccb16a6 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:03.619647 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:03.622341 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:03.622938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:03.623457 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:03.632546 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:03.738030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:03.768490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:03.770037 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:03.784606 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1409/2773] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:03 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:49:11.618619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:11.619559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:11.619559 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:11.671554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:11.673596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:11.701436 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:11.773102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1365/2774] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:49:11 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:49:11.845334 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:11.847668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:11.933053 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb68644-64a0-41a5-9459-d1f6b9d60928 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:11.940846 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:11.941886 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:11.942253 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:11.942715 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:11.953030 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:11.954704 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:11.965227 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1410/2775] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:49:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:49:21.760105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=14 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:21.960962 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=41 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:22.040381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:22.044797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:22.221859 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a328b3-2a8b-4402-80a2-fcc65b2ecb70 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:22.232285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:22.233701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:22.234184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:22.234702 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] PATH_INFO: `/v3/users/e04c679752fb459f84ac794a646d241a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:22.236574 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e04c679752fb459f84ac794a646d241a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:22.239637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:22.349846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e04c679752fb459f84ac794a646d241a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:22.369153 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e04c679752fb459f84ac794a646d241a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:22.387105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e04c679752fb459f84ac794a646d241a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:22.388197 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e04c679752fb459f84ac794a646d241a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:22.423054 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e04c679752fb459f84ac794a646d241a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:22.439013 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1366/2776] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:22 2022] DELETE /identity/v3/users/e04c679752fb459f84ac794a646d241a => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:22.827184 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:22.841403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05108eb-102f-4fc9-8953-40b919b3110a service swift] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:22.952887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:22.954345 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:22.957977 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:22.958795 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] PATH_INFO: `/v3/projects/0e8b934cdedf4654bb82f62880293b7f` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:22.961390 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e8b934cdedf4654bb82f62880293b7f)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:22.964759 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:23.065421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e8b934cdedf4654bb82f62880293b7f', 'request_id': 'req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.080596 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e8b934cdedf4654bb82f62880293b7f', 'request_id': 'req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.089724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e8b934cdedf4654bb82f62880293b7f', 'request_id': 'req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.106286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] The token cache is being invalidate because project 0e8b934cdedf4654bb82f62880293b7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:23.106905 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.108613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.111652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1411/2777] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:22 2022] DELETE /identity/v3/projects/0e8b934cdedf4654bb82f62880293b7f => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:23.115883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:23.118250 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:23.136956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d88c795-14b5-4e89-8a0f-d67143947e06 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:23.225629 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:23.227037 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:23.227493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:23.227965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] PATH_INFO: `/v3/users/fbfcc3bd9fba4c0aa7b9d9df3cae51e8` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:23.230159 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbfcc3bd9fba4c0aa7b9d9df3cae51e8)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:23.234090 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:23.279021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:23.319146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbfcc3bd9fba4c0aa7b9d9df3cae51e8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.328344 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbfcc3bd9fba4c0aa7b9d9df3cae51e8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.340357 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbfcc3bd9fba4c0aa7b9d9df3cae51e8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.342266 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbfcc3bd9fba4c0aa7b9d9df3cae51e8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.367539 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-39bd1617-05ca-448d-8221-23ac69227916 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbfcc3bd9fba4c0aa7b9d9df3cae51e8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.380179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1367/2778] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:23 2022] DELETE /identity/v3/users/fbfcc3bd9fba4c0aa7b9d9df3cae51e8 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:23.684325 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:23.690629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:23.721101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddcf029-0f42-4310-9bcf-2ae447d4e8cb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:23.845719 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:23.846898 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:23.847257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:23.847611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] PATH_INFO: `/v3/projects/dbd46346d1d24fbda86f76a3873a503e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:23.849136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbd46346d1d24fbda86f76a3873a503e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:23.851261 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:23.923683 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:23.953453 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbd46346d1d24fbda86f76a3873a503e', 'request_id': 'req-f8fa878e-458d-4396-a5fc-371b6445d5fc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.971179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbd46346d1d24fbda86f76a3873a503e', 'request_id': 'req-f8fa878e-458d-4396-a5fc-371b6445d5fc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:23.984179 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbd46346d1d24fbda86f76a3873a503e', 'request_id': 'req-f8fa878e-458d-4396-a5fc-371b6445d5fc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:24.007476 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] The token cache is being invalidate because project dbd46346d1d24fbda86f76a3873a503e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:24.007816 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:24.008908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f8fa878e-458d-4396-a5fc-371b6445d5fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:24.017046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1412/2779] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:23 2022] DELETE /identity/v3/projects/dbd46346d1d24fbda86f76a3873a503e => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:24.019994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3605ae7e-3111-4783-b2c5-21542864871e None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.020612 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3605ae7e-3111-4783-b2c5-21542864871e None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.021572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3605ae7e-3111-4783-b2c5-21542864871e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.061985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-3605ae7e-3111-4783-b2c5-21542864871e None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:24.064476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3605ae7e-3111-4783-b2c5-21542864871e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:24.192461 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1368/2780] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:24.203542 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cde6fa-7c85-4f81-83c6-e5f448671498 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.203973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cde6fa-7c85-4f81-83c6-e5f448671498 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.204423 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cde6fa-7c85-4f81-83c6-e5f448671498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.235910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-81cde6fa-7c85-4f81-83c6-e5f448671498 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:24.237837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-81cde6fa-7c85-4f81-83c6-e5f448671498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:24.251935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1413/2781] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:24.257659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3605ae7e-3111-4783-b2c5-21542864871e None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:24.273409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3605ae7e-3111-4783-b2c5-21542864871e None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:24.280897 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:24.281832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.282192 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.282608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.283409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:49:24.283823 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:24.285504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:24.309257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1369/2782] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:49:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 11 14:49:24.315525 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cde6fa-7c85-4f81-83c6-e5f448671498 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:24.327219 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cde6fa-7c85-4f81-83c6-e5f448671498 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:24.332737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:24.333973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.334136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.334516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.335318 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:24.337306 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:24.382273 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1414/2783] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:49:24 2022] POST /identity/v3/projects => generated 361 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:24.388113 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:24.414177 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28384d25-3939-48b6-a82b-b686f693c13d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:24.537255 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:24.539152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.539725 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.540598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.542600 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:24.546464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:24.596580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1370/2784] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:49:24 2022] POST /identity/v3/users => generated 281 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:24.598257 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:24.617272 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3681c413-5e51-41bb-911c-02cbe98a577a admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:24.626039 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:24.626772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.627070 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.627370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.628156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:24.632152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:24.650462 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1415/2785] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:24 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:24.656149 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:24.673898 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccf1f6d-de64-4f4d-adda-a5c750e1b920 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:24.684395 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b4813b-74d6-456f-8a69-fa9548027554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:24.685508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4813b-74d6-456f-8a69-fa9548027554 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.686623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4813b-74d6-456f-8a69-fa9548027554 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.687115 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4813b-74d6-456f-8a69-fa9548027554 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.688214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b4813b-74d6-456f-8a69-fa9548027554 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:24.690273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b4813b-74d6-456f-8a69-fa9548027554 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:24.711182 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1371/2786] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:24 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:49:24.715616 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:24.733934 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a808b5-5f67-4c7d-b854-af475992a5c7 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:24.741265 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:24.742326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.742735 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.743232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] PATH_INFO: `/v3/projects/ee0155f148c24db4863843ce41bb4754/users/dc4e57c8e5064a889c40cb3cc4fc802a/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.779213 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee0155f148c24db4863843ce41bb4754, user_id=dc4e57c8e5064a889c40cb3cc4fc802a, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:24.785126 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:24.829017 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1416/2787] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:24 2022] PUT /identity/v3/projects/ee0155f148c24db4863843ce41bb4754/users/dc4e57c8e5064a889c40cb3cc4fc802a/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:24.831095 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d08e1bc-2b9b-4a77-a794-218227bdc414 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:24.831623 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d08e1bc-2b9b-4a77-a794-218227bdc414 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:24.832126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d08e1bc-2b9b-4a77-a794-218227bdc414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:24.846003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:24.882141 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1d08e1bc-2b9b-4a77-a794-218227bdc414 None None] MFA Rules not processed for user `dc4e57c8e5064a889c40cb3cc4fc802a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:24.883662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1d08e1bc-2b9b-4a77-a794-218227bdc414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:24.985697 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1372/2788] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:24 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:49:25.038803 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:25.041751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:25.172856 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd780e-514b-43c6-9f4e-2575a235ed48 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:25.176561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0ee2b1-2395-4848-a9ef-e6098fcdc7d7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:25.177231 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0ee2b1-2395-4848-a9ef-e6098fcdc7d7 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:25.177515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0ee2b1-2395-4848-a9ef-e6098fcdc7d7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:25.177961 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0ee2b1-2395-4848-a9ef-e6098fcdc7d7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:25.187681 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0ee2b1-2395-4848-a9ef-e6098fcdc7d7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:25.189152 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0ee2b1-2395-4848-a9ef-e6098fcdc7d7 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:25.203140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1417/2789] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:25 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:49:25.219899 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d08e1bc-2b9b-4a77-a794-218227bdc414 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:25.331414 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d08e1bc-2b9b-4a77-a794-218227bdc414 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:25.338061 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:25.339297 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:25.339693 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:25.340124 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] PATH_INFO: `/v3/users/dc4e57c8e5064a889c40cb3cc4fc802a` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:25.341691 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc4e57c8e5064a889c40cb3cc4fc802a)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:25.344480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:25.422713 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc4e57c8e5064a889c40cb3cc4fc802a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:25.435706 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc4e57c8e5064a889c40cb3cc4fc802a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:25.453016 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc4e57c8e5064a889c40cb3cc4fc802a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:25.456955 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc4e57c8e5064a889c40cb3cc4fc802a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:25.501939 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc4e57c8e5064a889c40cb3cc4fc802a'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:25.513191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1373/2790] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:25 2022] DELETE /identity/v3/users/dc4e57c8e5064a889c40cb3cc4fc802a => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:49:25.518570 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:25.518879 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:25.519048 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:25.562714 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:25.564709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:25.719066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:25.743567 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1418/2791] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:25.769081 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:25.771659 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:25.803720 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:25.931573 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1616b9-6513-4ef4-9470-ebd404cb2dab admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:25.937615 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73713397-9989-45d4-ab80-bb38417cb669 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:25.938566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73713397-9989-45d4-ab80-bb38417cb669 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:25.938949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73713397-9989-45d4-ab80-bb38417cb669 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:25.939275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73713397-9989-45d4-ab80-bb38417cb669 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:25.950795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73713397-9989-45d4-ab80-bb38417cb669 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:25.954388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73713397-9989-45d4-ab80-bb38417cb669 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:25.977795 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1374/2792] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:49:26.465404 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:26.468553 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:26.528380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2667fed-d65c-49fd-a294-d0dfb67974fd None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:26.536268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:26.537632 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:26.538055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:26.538458 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] PATH_INFO: `/v3/projects/ee0155f148c24db4863843ce41bb4754` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:26.540749 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee0155f148c24db4863843ce41bb4754)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:26.543279 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:26.639854 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee0155f148c24db4863843ce41bb4754', 'request_id': 'req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:26.654696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee0155f148c24db4863843ce41bb4754', 'request_id': 'req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:26.687140 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee0155f148c24db4863843ce41bb4754', 'request_id': 'req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:26.733705 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] The token cache is being invalidate because project ee0155f148c24db4863843ce41bb4754 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:26.734614 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:26.737218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-e374a04f-36ad-4aaa-aaf7-3aa9484c3b49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:26.746580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1419/2793] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:26 2022] DELETE /identity/v3/projects/ee0155f148c24db4863843ce41bb4754 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:26.750056 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3923086-5fec-4eff-8d7e-127884811481 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:26.751434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3923086-5fec-4eff-8d7e-127884811481 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:26.751434 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3923086-5fec-4eff-8d7e-127884811481 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:26.786086 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-c3923086-5fec-4eff-8d7e-127884811481 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:26.788312 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-c3923086-5fec-4eff-8d7e-127884811481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:26.926524 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1375/2794] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:27.066275 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf9c2a-5f89-485f-9cbf-fdc8252fc0f9 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.067805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf9c2a-5f89-485f-9cbf-fdc8252fc0f9 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.071829 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf9c2a-5f89-485f-9cbf-fdc8252fc0f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.132388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-edbf9c2a-5f89-485f-9cbf-fdc8252fc0f9 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:27.135488 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-edbf9c2a-5f89-485f-9cbf-fdc8252fc0f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:27.163582 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1420/2795] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:27.171358 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3923086-5fec-4eff-8d7e-127884811481 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:27.186207 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3923086-5fec-4eff-8d7e-127884811481 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:27.192507 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:27.193241 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.193668 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.193862 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.194616 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:49:27.194962 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:27.196882 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:27.212024 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1376/2796] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:49:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:49:27.217956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf9c2a-5f89-485f-9cbf-fdc8252fc0f9 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:27.242351 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf9c2a-5f89-485f-9cbf-fdc8252fc0f9 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:27.254587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:27.256008 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.256581 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.257134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.259210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:27.264081 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:27.307211 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1421/2797] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:49:27 2022] POST /identity/v3/projects => generated 385 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:27.308028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:27.339025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c449f19-38bb-4bd0-8285-a0ad6402ae1b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:27.470503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:27.471647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.471918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.472364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.473360 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:27.476158 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:27.502845 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1377/2798] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:49:27 2022] POST /identity/v3/users => generated 293 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:27.508108 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:27.530613 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff95999a-550c-415e-b0c9-610edde19472 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:27.540739 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:27.542649 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.543667 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.544539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.546891 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:27.550869 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:27.576455 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1422/2799] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:27 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:27.581063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:27.603382 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8462ef1c-b15b-402b-a6f1-af114b9e8118 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:27.612967 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d4d804-1e0d-4a17-a632-6ac20397635e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:27.614922 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d4d804-1e0d-4a17-a632-6ac20397635e admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.615832 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d4d804-1e0d-4a17-a632-6ac20397635e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.616738 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d4d804-1e0d-4a17-a632-6ac20397635e admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.618596 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d4d804-1e0d-4a17-a632-6ac20397635e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:27.622187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d4d804-1e0d-4a17-a632-6ac20397635e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:27.649814 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1378/2800] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:27 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:27.655908 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:27.668084 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b130a90e-c7c2-426f-9fb8-381b56d8d391 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:27.673427 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:27.674369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.674696 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.674965 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] PATH_INFO: `/v3/projects/32a542ab99854d55b6f56d78de0f4a14/users/12255457cd924e50a559b8eb2f106c77/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.694218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32a542ab99854d55b6f56d78de0f4a14, user_id=12255457cd924e50a559b8eb2f106c77, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:27.697820 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:27.718406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1423/2801] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:27 2022] PUT /identity/v3/projects/32a542ab99854d55b6f56d78de0f4a14/users/12255457cd924e50a559b8eb2f106c77/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:27.730804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ecd0f-2409-46f7-a40e-7ff103d02423 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:27.731211 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ecd0f-2409-46f7-a40e-7ff103d02423 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:27.731460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ecd0f-2409-46f7-a40e-7ff103d02423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:27.768285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-9f9ecd0f-2409-46f7-a40e-7ff103d02423 None None] MFA Rules not processed for user `12255457cd924e50a559b8eb2f106c77`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:27.769840 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-9f9ecd0f-2409-46f7-a40e-7ff103d02423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:27.782523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:27.893520 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1379/2802] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:27 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:49:27.898415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:28.027949 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364016ca-818b-4f90-a3da-a28fa38d5606 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:28.042669 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:28.044167 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:28.044587 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:28.045121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:28.046271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:28.049804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:28.102206 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1424/2803] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:49:27 2022] POST /identity/v3/projects => generated 385 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:28.108995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ecd0f-2409-46f7-a40e-7ff103d02423 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:28.124943 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ecd0f-2409-46f7-a40e-7ff103d02423 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:28.255025 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:28.256460 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:28.257101 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:28.257861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:28.259608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:28.262879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:28.301527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1380/2804] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:49:28 2022] POST /identity/v3/users => generated 293 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:49:28.306657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:28.319918 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:28.327366 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbc0791-241f-49b4-8d51-82f86ba30db6 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:28.349543 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:28.350939 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:28.351552 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:28.352148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:28.358636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:28.358636 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:28.382611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1425/2805] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:28 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:28.387918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:28.409902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cad3ef-e165-4a96-83ed-94aac32dbefa admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:28.418632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96780da7-5e26-4e64-9e20-a795d08fea9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:28.419998 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96780da7-5e26-4e64-9e20-a795d08fea9d admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:28.420562 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96780da7-5e26-4e64-9e20-a795d08fea9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:28.421139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96780da7-5e26-4e64-9e20-a795d08fea9d admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:28.422731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96780da7-5e26-4e64-9e20-a795d08fea9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:28.426121 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96780da7-5e26-4e64-9e20-a795d08fea9d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:28.445039 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1381/2806] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:28 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:28.450093 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:28.468493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b6676-21b0-4dd8-b532-23cef55b04b3 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:28.474709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:28.476032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:28.476629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:28.477246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] PATH_INFO: `/v3/projects/856dfcf5095945479bfce3145d6aec3a/users/725214e932924a82a1d6b52666d04c60/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:28.502566 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] RBAC: Authorizing `identity:create_grant(project_id=856dfcf5095945479bfce3145d6aec3a, user_id=725214e932924a82a1d6b52666d04c60, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:28.505532 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:28.525629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1426/2807] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:28 2022] PUT /identity/v3/projects/856dfcf5095945479bfce3145d6aec3a/users/725214e932924a82a1d6b52666d04c60/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:28.543658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc5a0b-3a37-43c8-87dd-8faae15319bf None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:28.543965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc5a0b-3a37-43c8-87dd-8faae15319bf None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:28.544245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc5a0b-3a37-43c8-87dd-8faae15319bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:28.587033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-08fc5a0b-3a37-43c8-87dd-8faae15319bf None None] MFA Rules not processed for user `725214e932924a82a1d6b52666d04c60`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:28.588733 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-08fc5a0b-3a37-43c8-87dd-8faae15319bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:28.713686 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1382/2808] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:49:28.721377 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:28.830651 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2b8561-a3cf-4ab1-a85d-e5af669d5580 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:28.836840 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:28.837593 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:28.838074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:28.838367 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:28.839197 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:28.841425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:28.900469 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1427/2809] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:49:28 2022] POST /identity/v3/projects => generated 383 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:28.904560 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fc5a0b-3a37-43c8-87dd-8faae15319bf None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:28.926078 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fc5a0b-3a37-43c8-87dd-8faae15319bf None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:28.965142 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:29.069167 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:29.070381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:29.070944 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:29.071483 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:29.073021 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:29.077071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:29.113587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1383/2810] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:49:28 2022] POST /identity/v3/users => generated 298 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:29.119130 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:29.138577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e9b747-6590-48ed-96e6-92d89bcf2bed admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:29.146936 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:29.148402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:29.148919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:29.149421 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:29.150907 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:29.154078 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:29.174222 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1428/2811] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:29 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:29.179063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:29.191332 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a017aa-c766-48b0-832f-9e0c54abd21e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:29.198993 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:29.200724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:29.201476 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:29.201912 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] PATH_INFO: `/v3/projects/3b31c09244b84d9daa8efa10aec12a26/users/5d01f17089a6420895b1ff39b8c039dd/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:29.231643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b31c09244b84d9daa8efa10aec12a26, user_id=5d01f17089a6420895b1ff39b8c039dd, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:29.239551 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:29.263729 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1384/2812] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:29 2022] PUT /identity/v3/projects/3b31c09244b84d9daa8efa10aec12a26/users/5d01f17089a6420895b1ff39b8c039dd/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:29.268755 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:29.290182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60292b3d-294e-4771-b154-8ed981524b76 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:29.412246 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fe4869-cc47-4c3c-81ce-a7488f3eed6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:29.413646 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fe4869-cc47-4c3c-81ce-a7488f3eed6c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:29.414204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fe4869-cc47-4c3c-81ce-a7488f3eed6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:29.414805 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fe4869-cc47-4c3c-81ce-a7488f3eed6c admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:29.416099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fe4869-cc47-4c3c-81ce-a7488f3eed6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:29.418725 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fe4869-cc47-4c3c-81ce-a7488f3eed6c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:29.444776 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1429/2813] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:29 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:49:29.447053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:29.467689 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aaf322-7668-4a50-b82c-e4b7a12490f1 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:29.476760 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:29.478542 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:29.479091 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:29.479678 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] PATH_INFO: `/v3/projects/3b31c09244b84d9daa8efa10aec12a26/users/5d01f17089a6420895b1ff39b8c039dd/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:29.484302 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b31c09244b84d9daa8efa10aec12a26, user_id=5d01f17089a6420895b1ff39b8c039dd, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:29.489746 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:29.512947 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1385/2814] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:29 2022] PUT /identity/v3/projects/3b31c09244b84d9daa8efa10aec12a26/users/5d01f17089a6420895b1ff39b8c039dd/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:49:29.546117 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60c5f9-c550-4e89-8d97-16ce1692aac8 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:29.547056 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60c5f9-c550-4e89-8d97-16ce1692aac8 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:29.547376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60c5f9-c550-4e89-8d97-16ce1692aac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:29.587995 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-2a60c5f9-c550-4e89-8d97-16ce1692aac8 None None] MFA Rules not processed for user `5d01f17089a6420895b1ff39b8c039dd`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:29.589408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2a60c5f9-c550-4e89-8d97-16ce1692aac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:29.638086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:29.702899 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1430/2815] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:29 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:49:29.718777 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:29.720664 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:29.802872 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99909d59-f226-4185-aa44-d6b4ee03171c admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:29.807579 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:29.808183 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:29.808467 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:29.808783 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:29.881990 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:29.883502 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:29.894982 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1386/2816] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:49:30.407580 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a2145-57ce-49c7-b4f9-6ee175766c2a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:30.408007 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a2145-57ce-49c7-b4f9-6ee175766c2a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:30.408380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a2145-57ce-49c7-b4f9-6ee175766c2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:30.429216 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-b71a2145-57ce-49c7-b4f9-6ee175766c2a None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:30.430598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b71a2145-57ce-49c7-b4f9-6ee175766c2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:30.497848 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1431/2817] 10.222.0.72 () {58 vars in 967 bytes} [Mon Jul 11 14:49:30 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:30.514950 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:30.516655 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:30.560243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:30.613828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c8e9d5-84f0-48e3-ba36-d4f2c884ce75 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:30.620139 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:30.621023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:30.621363 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:30.621690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:30.632870 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:30.634930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:30.644951 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1387/2818] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:49:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:49:31.519090 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a2145-57ce-49c7-b4f9-6ee175766c2a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:31.531807 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a2145-57ce-49c7-b4f9-6ee175766c2a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:31.536212 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:31.536885 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:31.537169 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:31.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:31.610783 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:31.612493 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:31.624598 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1432/2819] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:31 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:49:32.312001 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:32.323173 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009f0ca7-3bf8-43b6-8422-8117ec44fa13 service swift] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:32.327552 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:32.328281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:32.328593 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:32.328956 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:32.340792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:32.343510 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:32.354825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1388/2820] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:32 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:49:32.784214 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:32.786758 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:32.876300 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590d813a-b16e-40b9-b975-b4b4bd923ca4 service cinder] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:32.881534 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:32.882470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:32.882754 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:32.883032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] PATH_INFO: `/v3/users/6e9ce7b0fdbd43f28d38a4b5657a3e2e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:32.884657 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e9ce7b0fdbd43f28d38a4b5657a3e2e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:32.887544 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:32.900361 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=100 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:32.956523 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e9ce7b0fdbd43f28d38a4b5657a3e2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:32.966405 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e9ce7b0fdbd43f28d38a4b5657a3e2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:32.967157 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e9ce7b0fdbd43f28d38a4b5657a3e2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:32.977026 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e9ce7b0fdbd43f28d38a4b5657a3e2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:32.993590 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-f72c3f7e-405e-406c-b83c-f8d3089ad106 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e9ce7b0fdbd43f28d38a4b5657a3e2e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:33.001016 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1433/2821] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:32 2022] DELETE /identity/v3/users/6e9ce7b0fdbd43f28d38a4b5657a3e2e => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:33.238935 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:33.253033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364a3080-c06e-4e74-bf89-8acaef9ff0e8 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:33.330371 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:33.331317 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.331825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.332125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] PATH_INFO: `/v3/projects/23a54d0a3442474db1829cb42e2b3c95` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.333726 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23a54d0a3442474db1829cb42e2b3c95)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:33.335739 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:33.392986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23a54d0a3442474db1829cb42e2b3c95', 'request_id': 'req-1d5489f2-9611-4d37-a892-5244766a8ef8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:33.399757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23a54d0a3442474db1829cb42e2b3c95', 'request_id': 'req-1d5489f2-9611-4d37-a892-5244766a8ef8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:33.410557 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23a54d0a3442474db1829cb42e2b3c95', 'request_id': 'req-1d5489f2-9611-4d37-a892-5244766a8ef8'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:33.424804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] The token cache is being invalidate because project 23a54d0a3442474db1829cb42e2b3c95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:33.425099 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:33.425786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-1d5489f2-9611-4d37-a892-5244766a8ef8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:33.427425 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1389/2822] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:33 2022] DELETE /identity/v3/projects/23a54d0a3442474db1829cb42e2b3c95 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:33.432906 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28fba54-fcfd-4b15-aed5-b1e76d21e4f4 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.433263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28fba54-fcfd-4b15-aed5-b1e76d21e4f4 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.433594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28fba54-fcfd-4b15-aed5-b1e76d21e4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.454815 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e28fba54-fcfd-4b15-aed5-b1e76d21e4f4 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:33.456102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e28fba54-fcfd-4b15-aed5-b1e76d21e4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:33.538470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1434/2823] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:33.550285 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b64c65-e29b-4be5-b3c1-def4b3b5c2f9 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.550480 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b64c65-e29b-4be5-b3c1-def4b3b5c2f9 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.550800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b64c65-e29b-4be5-b3c1-def4b3b5c2f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.571445 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-16b64c65-e29b-4be5-b3c1-def4b3b5c2f9 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:33.572868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-16b64c65-e29b-4be5-b3c1-def4b3b5c2f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:33.582536 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1390/2824] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:33.587449 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28fba54-fcfd-4b15-aed5-b1e76d21e4f4 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:33.598662 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28fba54-fcfd-4b15-aed5-b1e76d21e4f4 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:33.603855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:33.604548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.604836 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.605136 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] PATH_INFO: `/v3/domains` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.605887 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:49:33.606203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:33.607982 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:33.627743 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1435/2825] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:49:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:49:33.628984 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b64c65-e29b-4be5-b3c1-def4b3b5c2f9 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:33.639485 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b64c65-e29b-4be5-b3c1-def4b3b5c2f9 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:33.646780 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:33.647464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.647952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.648252 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] PATH_INFO: `/v3/projects` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.648949 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:33.650942 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:33.686337 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1391/2826] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:49:33 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:33.690823 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:33.702717 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4e5a15-23aa-4d7d-854a-7b8d00350e6d admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:33.796416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:33.797912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.798387 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.799087 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] PATH_INFO: `/v3/users` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.800221 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:33.802884 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:33.832730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1436/2827] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:49:33 2022] POST /identity/v3/users => generated 287 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:33.837639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:33.845639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:33.850530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459d4ed5-f461-46a2-9799-5a9031875018 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:33.856817 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:33.857639 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.858292 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.858868 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.859977 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:33.861767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:33.876486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1392/2828] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:33.882966 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:33.894403 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:33.899252 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c91f98-fe77-4f45-a026-e068d0041203 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:33.909526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a4180-eb22-4d1e-91f4-acd0d504edbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:33.910305 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a4180-eb22-4d1e-91f4-acd0d504edbc admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.910633 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a4180-eb22-4d1e-91f4-acd0d504edbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.910919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a4180-eb22-4d1e-91f4-acd0d504edbc admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.911684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676a4180-eb22-4d1e-91f4-acd0d504edbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:33.913682 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676a4180-eb22-4d1e-91f4-acd0d504edbc admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:33.925992 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1437/2829] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:33 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:33.930800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:33.943413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe0e496-ece5-41fc-a57e-17a9475e7883 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:33.948808 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:33.949731 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.950035 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.950314 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] PATH_INFO: `/v3/projects/c467ecb2524b41e4840d81dd2128396a/users/47a0b93509ee497e98c3db9947c16e80/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:33.968333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c467ecb2524b41e4840d81dd2128396a, user_id=47a0b93509ee497e98c3db9947c16e80, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:33.971530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:33.986319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1393/2830] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:33 2022] PUT /identity/v3/projects/c467ecb2524b41e4840d81dd2128396a/users/47a0b93509ee497e98c3db9947c16e80/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:33.997481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed7c76-7522-4c0f-966c-62d8d348895d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:33.998313 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed7c76-7522-4c0f-966c-62d8d348895d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:33.998624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed7c76-7522-4c0f-966c-62d8d348895d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:34.032229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e2ed7c76-7522-4c0f-966c-62d8d348895d None None] MFA Rules not processed for user `47a0b93509ee497e98c3db9947c16e80`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:34.033821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e2ed7c76-7522-4c0f-966c-62d8d348895d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:34.116757 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1438/2831] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:34.130568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:34.133535 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:34.220464 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31c12c0-168b-47d1-ad17-a579fb67f95e admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:34.226822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4f1b0819d52d481898b354a5ae182746', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:34.227928 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:34.228425 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:34.228903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:34.244307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:34.246377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:34.260193 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1394/2832] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:34 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:49:35.661243 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b220b42-90e1-4fd8-a490-081fc013d19d None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:35.662133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b220b42-90e1-4fd8-a490-081fc013d19d None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:35.662133 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b220b42-90e1-4fd8-a490-081fc013d19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:35.683329 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-1b220b42-90e1-4fd8-a490-081fc013d19d None None] MFA Rules not processed for user `47a0b93509ee497e98c3db9947c16e80`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:35.684602 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1b220b42-90e1-4fd8-a490-081fc013d19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:35.691763 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1439/2833] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:35 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:36.431487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:36.434540 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:36.555245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7477c3c-3c15-432f-a639-e8d49a95f329 service cinder] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:36.561087 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:36.562175 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:36.562622 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:36.563003 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] PATH_INFO: `/v3/users/12255457cd924e50a559b8eb2f106c77` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:36.564591 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12255457cd924e50a559b8eb2f106c77)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:36.567398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:36.637259 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12255457cd924e50a559b8eb2f106c77'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:36.647240 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12255457cd924e50a559b8eb2f106c77'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:36.660097 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12255457cd924e50a559b8eb2f106c77'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:36.661190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12255457cd924e50a559b8eb2f106c77'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:36.686068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:36.696233 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12255457cd924e50a559b8eb2f106c77'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:36.708757 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1395/2834] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:36 2022] DELETE /identity/v3/users/12255457cd924e50a559b8eb2f106c77 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:36.714388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:36.715269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:36.715269 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:36.741594 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:36.743875 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:36.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1440/2835] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:36.842386 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:36.844619 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:36.925756 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a804d4-e44d-4465-b764-a861c354c659 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:36.930797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:36.931662 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:36.931994 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:36.932341 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:36.945407 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:36.947257 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:36.962074 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1396/2836] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:49:37.372200 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:37.374529 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:37.400738 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8692edb-44dc-4cdb-a90b-fe63326c269a None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:37.417984 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:37.418972 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:37.419270 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:37.420089 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] PATH_INFO: `/v3/projects/32a542ab99854d55b6f56d78de0f4a14` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:37.421637 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=32a542ab99854d55b6f56d78de0f4a14)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:37.424072 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:37.507210 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32a542ab99854d55b6f56d78de0f4a14', 'request_id': 'req-af955c16-0d77-4560-9b23-879a6e351c9f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.520310 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32a542ab99854d55b6f56d78de0f4a14', 'request_id': 'req-af955c16-0d77-4560-9b23-879a6e351c9f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.543737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32a542ab99854d55b6f56d78de0f4a14', 'request_id': 'req-af955c16-0d77-4560-9b23-879a6e351c9f'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.557685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:37.563286 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] The token cache is being invalidate because project 32a542ab99854d55b6f56d78de0f4a14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:37.563676 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.564766 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.567063 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1441/2837] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:37 2022] DELETE /identity/v3/projects/32a542ab99854d55b6f56d78de0f4a14 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:37.571033 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:37.584749 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c639c2-053f-41f5-9c0b-da869a57ad5b service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:37.660355 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:37.661264 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:37.661563 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:37.661825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] PATH_INFO: `/v3/users/725214e932924a82a1d6b52666d04c60` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:37.663187 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=725214e932924a82a1d6b52666d04c60)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:37.665168 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:37.721498 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '725214e932924a82a1d6b52666d04c60'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.729283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '725214e932924a82a1d6b52666d04c60'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.738902 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '725214e932924a82a1d6b52666d04c60'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.739554 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '725214e932924a82a1d6b52666d04c60'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.754690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '725214e932924a82a1d6b52666d04c60'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:37.761991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1397/2838] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:37 2022] DELETE /identity/v3/users/725214e932924a82a1d6b52666d04c60 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:49:38.130771 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:38.134825 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:38.157102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af955c16-0d77-4560-9b23-879a6e351c9f admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:38.228685 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:38.229702 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:38.230465 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:38.230921 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] PATH_INFO: `/v3/projects/856dfcf5095945479bfce3145d6aec3a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:38.233578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=856dfcf5095945479bfce3145d6aec3a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:38.236466 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:38.299283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '856dfcf5095945479bfce3145d6aec3a', 'request_id': 'req-34472ca1-c8d3-48b0-a794-cb6ea198fac0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.309548 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '856dfcf5095945479bfce3145d6aec3a', 'request_id': 'req-34472ca1-c8d3-48b0-a794-cb6ea198fac0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.318349 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '856dfcf5095945479bfce3145d6aec3a', 'request_id': 'req-34472ca1-c8d3-48b0-a794-cb6ea198fac0'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.331232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] The token cache is being invalidate because project 856dfcf5095945479bfce3145d6aec3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:38.331531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.332232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.334487 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1442/2839] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:38 2022] DELETE /identity/v3/projects/856dfcf5095945479bfce3145d6aec3a => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:38.338851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:38.340427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:38.367577 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3d45d-c4be-407c-a176-638e93abae8b admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:38.379263 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:38.443015 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:38.444017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:38.444281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:38.444565 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] PATH_INFO: `/v3/users/5d01f17089a6420895b1ff39b8c039dd` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:38.446040 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d01f17089a6420895b1ff39b8c039dd)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:38.448103 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:38.519398 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d01f17089a6420895b1ff39b8c039dd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.533683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d01f17089a6420895b1ff39b8c039dd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.558543 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d01f17089a6420895b1ff39b8c039dd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.559903 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d01f17089a6420895b1ff39b8c039dd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.582272 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-9612aa41-6c84-4934-884f-5ad344e6e843 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d01f17089a6420895b1ff39b8c039dd'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:38.594171 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1398/2840] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:38 2022] DELETE /identity/v3/users/5d01f17089a6420895b1ff39b8c039dd => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:39.211218 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:39.212812 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:39.227911 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34472ca1-c8d3-48b0-a794-cb6ea198fac0 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:39.312138 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:39.314018 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:39.314675 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:39.315186 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] PATH_INFO: `/v3/projects/3b31c09244b84d9daa8efa10aec12a26` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:39.317956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b31c09244b84d9daa8efa10aec12a26)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:39.322088 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:39.383400 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:39.412203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b31c09244b84d9daa8efa10aec12a26', 'request_id': 'req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:39.425264 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b31c09244b84d9daa8efa10aec12a26', 'request_id': 'req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:39.435388 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b31c09244b84d9daa8efa10aec12a26', 'request_id': 'req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:39.451065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] The token cache is being invalidate because project 3b31c09244b84d9daa8efa10aec12a26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:39.451415 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:39.452066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:39.453892 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1443/2841] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:39 2022] DELETE /identity/v3/projects/3b31c09244b84d9daa8efa10aec12a26 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:48.586415 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=47,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:49.447074 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:51.430012 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cee8aa-b788-4d5f-a459-f82b48b46380 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:51.431018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cee8aa-b788-4d5f-a459-f82b48b46380 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:51.431018 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cee8aa-b788-4d5f-a459-f82b48b46380 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:51.452053 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-99cee8aa-b788-4d5f-a459-f82b48b46380 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:51.453527 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-99cee8aa-b788-4d5f-a459-f82b48b46380 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:51.545112 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1399/2842] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:49:51 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:51.559195 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:51.562122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:51.660611 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b055a3-5e3b-45a0-9033-0d1b6f35ec0c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:51.669477 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:51.670322 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:51.670850 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:51.671147 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:51.683229 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:51.685785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:51.696452 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1444/2843] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:49:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:49:52.406828 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d182d6-60aa-47d5-894d-bebfef3ad477 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:52.407261 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d182d6-60aa-47d5-894d-bebfef3ad477 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:52.407761 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d182d6-60aa-47d5-894d-bebfef3ad477 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:52.430847 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-91d182d6-60aa-47d5-894d-bebfef3ad477 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:52.432724 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-91d182d6-60aa-47d5-894d-bebfef3ad477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:52.441110 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1400/2844] 10.222.0.72 () {58 vars in 967 bytes} [Mon Jul 11 14:49:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:52.456317 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:52.470480 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916368-f2ba-4d39-aad6-1a18faefe620 service swift] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:52.476440 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:52.477019 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:52.477307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:52.477584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:52.488802 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:52.490501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:52.503626 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1445/2845] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:49:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:49:52.775491 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4c1239-5729-4182-a514-4a306e9a7393 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:52.776044 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4c1239-5729-4182-a514-4a306e9a7393 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:52.776519 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4c1239-5729-4182-a514-4a306e9a7393 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:52.799093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-da4c1239-5729-4182-a514-4a306e9a7393 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:52.800388 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-da4c1239-5729-4182-a514-4a306e9a7393 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:52.808513 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1401/2846] 10.222.0.72 () {58 vars in 1009 bytes} [Mon Jul 11 14:49:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:52.822467 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:52.835864 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f8a6d4-fb22-46f1-a698-46d937f8fafe service swift] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:52.842769 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:52.843490 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:52.843920 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:52.844268 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:52.854577 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:52.856638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:52.868326 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1446/2847] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:49:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:49:53.339032 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:53.339741 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:53.340253 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:53.369478 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] MFA Rules not processed for user `647a67b450184f48b3d035fe875be47c`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:53.370985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:53.379550 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1402/2848] 10.222.0.72 () {58 vars in 967 bytes} [Mon Jul 11 14:49:53 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:53.395381 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:53.408658 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983282f-52e5-41ab-bf55-0174875e0db7 service swift] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:53.415010 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05091fc4-f221-4007-bd99-f9b9d3f0e21f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '055512e4e3744833ac6a9e61d22f0b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:53.415689 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05091fc4-f221-4007-bd99-f9b9d3f0e21f service swift] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:53.416036 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05091fc4-f221-4007-bd99-f9b9d3f0e21f service swift] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:53.416374 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05091fc4-f221-4007-bd99-f9b9d3f0e21f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:53.426592 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05091fc4-f221-4007-bd99-f9b9d3f0e21f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:53.428521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05091fc4-f221-4007-bd99-f9b9d3f0e21f service swift] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:53.439938 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1447/2849] 10.222.0.72 () {60 vars in 1384 bytes} [Mon Jul 11 14:49:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 11 14:49:56.022404 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:56.024041 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:56.115826 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1189f5d3-475c-4655-9842-6ed7dcd1c1f4 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:56.121009 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:56.122564 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:56.122887 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:56.123214 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] PATH_INFO: `/v3/users/47a0b93509ee497e98c3db9947c16e80` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:56.124604 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] RBAC: Authorizing `identity:delete_user(user_id=47a0b93509ee497e98c3db9947c16e80)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:56.126700 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:56.184028 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47a0b93509ee497e98c3db9947c16e80'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.192658 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47a0b93509ee497e98c3db9947c16e80'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.202740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47a0b93509ee497e98c3db9947c16e80'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.203448 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47a0b93509ee497e98c3db9947c16e80'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.219422 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47a0b93509ee497e98c3db9947c16e80'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.226334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1403/2850] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:49:56 2022] DELETE /identity/v3/users/47a0b93509ee497e98c3db9947c16e80 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:49:56.231406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:56.231963 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:56.232097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:56.255513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:56.257557 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:56.352201 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1448/2851] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:49:56.362797 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:56.364652 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:56.372861 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:56.437449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce57aa1-cd06-4398-909d-1efe7762938f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:56.442022 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee3b96c-e070-4868-9273-3c43815fa255 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:56.442504 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3b96c-e070-4868-9273-3c43815fa255 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:56.442905 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3b96c-e070-4868-9273-3c43815fa255 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:56.443191 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3b96c-e070-4868-9273-3c43815fa255 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:56.456155 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee3b96c-e070-4868-9273-3c43815fa255 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:56.458680 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee3b96c-e070-4868-9273-3c43815fa255 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:56.473351 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1404/2852] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:49:56.724539 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:56.727006 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:56.745358 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b159647-d4b7-4e6d-be60-766ebdc6d2d5 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:56.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:56.753625 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:56.754040 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:56.754571 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] PATH_INFO: `/v3/projects/c467ecb2524b41e4840d81dd2128396a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:56.757298 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c467ecb2524b41e4840d81dd2128396a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:56.761619 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:56.827929 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:56.831125 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c467ecb2524b41e4840d81dd2128396a', 'request_id': 'req-d4ba20a8-ffda-41de-9f37-78324a952b72'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.842640 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c467ecb2524b41e4840d81dd2128396a', 'request_id': 'req-d4ba20a8-ffda-41de-9f37-78324a952b72'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.850500 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c467ecb2524b41e4840d81dd2128396a', 'request_id': 'req-d4ba20a8-ffda-41de-9f37-78324a952b72'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.864730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] The token cache is being invalidate because project c467ecb2524b41e4840d81dd2128396a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:49:56.865031 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.865736 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-d4ba20a8-ffda-41de-9f37-78324a952b72 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:49:56.867406 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1449/2853] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:49:56 2022] DELETE /identity/v3/projects/c467ecb2524b41e4840d81dd2128396a => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:56.873784 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285cb6f-ce6a-48f7-b4c6-43921c03f1f0 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:56.874132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285cb6f-ce6a-48f7-b4c6-43921c03f1f0 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:56.874587 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1285cb6f-ce6a-48f7-b4c6-43921c03f1f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:56.894505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-1285cb6f-ce6a-48f7-b4c6-43921c03f1f0 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:56.895771 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-1285cb6f-ce6a-48f7-b4c6-43921c03f1f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:56.969128 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1405/2854] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:56.982481 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddde252-0bb5-4218-9a63-806342ff7d52 None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:56.983041 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddde252-0bb5-4218-9a63-806342ff7d52 None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:56.983424 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddde252-0bb5-4218-9a63-806342ff7d52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.015583 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-cddde252-0bb5-4218-9a63-806342ff7d52 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:57.017706 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-cddde252-0bb5-4218-9a63-806342ff7d52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:57.031668 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1450/2855] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:57.038132 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1285cb6f-ce6a-48f7-b4c6-43921c03f1f0 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:57.049127 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1285cb6f-ce6a-48f7-b4c6-43921c03f1f0 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:57.054376 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:57.055119 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.055409 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.055690 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] PATH_INFO: `/v3/domains` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.056450 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 11 14:49:57.056767 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:57.058500 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:57.082378 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1406/2856] 10.222.0.72 () {58 vars in 1152 bytes} [Mon Jul 11 14:49:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 11 14:49:57.088398 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddde252-0bb5-4218-9a63-806342ff7d52 None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:57.099380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddde252-0bb5-4218-9a63-806342ff7d52 None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:57.105498 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:57.106283 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.106724 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.107220 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.108501 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:57.110628 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:57.142513 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1451/2857] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:49:57 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 11 14:49:57.147135 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:57.158447 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a99a540-248a-4211-a7f5-970e559a4f81 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:57.221548 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:57.222243 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.222566 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.222853 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.223568 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:57.225544 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:57.248075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1407/2858] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:49:57 2022] POST /identity/v3/users => generated 287 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:49:57.252638 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:57.264225 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6489b6f8-e18c-441c-bfaf-0cb02af2dcbb admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:57.269451 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:57.270139 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.270491 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.270751 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.271508 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:57.273555 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:57.286274 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1452/2859] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:57.290979 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:57.302822 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28823b96-6054-47c2-9287-b59dbada671f admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:57.307802 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39105ea6-6cd8-4af9-9f9a-92ddfeec2c07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:57.308499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39105ea6-6cd8-4af9-9f9a-92ddfeec2c07 admin admin] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.308920 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39105ea6-6cd8-4af9-9f9a-92ddfeec2c07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.309204 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39105ea6-6cd8-4af9-9f9a-92ddfeec2c07 admin admin] PATH_INFO: `/v3/roles` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.310027 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39105ea6-6cd8-4af9-9f9a-92ddfeec2c07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:57.311716 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39105ea6-6cd8-4af9-9f9a-92ddfeec2c07 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:57.323036 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1408/2860] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:49:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:49:57.328373 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:57.340102 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8adb02-0dd3-41c9-9192-e04c26959c8c admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:57.345973 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:57.346925 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] REQUEST_METHOD: `PUT` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.347209 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.347485 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] PATH_INFO: `/v3/projects/6c2327c73cc445ba97c2ef24ef45e4d5/users/a7ecfb9570f746a0b1a9c29ff7cd64ef/roles/0d2588012d264232b288e3d89e4579c3` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.365113 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c2327c73cc445ba97c2ef24ef45e4d5, user_id=a7ecfb9570f746a0b1a9c29ff7cd64ef, role_id=0d2588012d264232b288e3d89e4579c3)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:57.368068 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:57.383919 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1453/2861] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:49:57 2022] PUT /identity/v3/projects/6c2327c73cc445ba97c2ef24ef45e4d5/users/a7ecfb9570f746a0b1a9c29ff7cd64ef/roles/0d2588012d264232b288e3d89e4579c3 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:49:57.389683 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad84433-f6c9-4932-bbb1-a4906e60ff89 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.389985 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad84433-f6c9-4932-bbb1-a4906e60ff89 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.390262 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad84433-f6c9-4932-bbb1-a4906e60ff89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.411486 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-4ad84433-f6c9-4932-bbb1-a4906e60ff89 None None] MFA Rules not processed for user `a7d8b4db44114e1ba544d78dc12ef30a`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:49:57.412841 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-4ad84433-f6c9-4932-bbb1-a4906e60ff89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:57.423353 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:49:57.491273 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1409/2862] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:49:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:49:57.502910 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:49:57.505603 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:49:57.577584 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812fc8ef-662a-452f-9c27-7bc76daef4ab admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:49:57.582450 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:49:57.583516 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:49:57.583896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:49:57.584287 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:49:57.596297 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:49:57.597701 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:49:57.608901 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1454/2863] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:49:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:50:02.946391 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889667-6d69-4da6-82fa-1a957297c371 None None] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:02.947071 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889667-6d69-4da6-82fa-1a957297c371 None None] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:02.947435 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07889667-6d69-4da6-82fa-1a957297c371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:03.000244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.auth.core [None req-07889667-6d69-4da6-82fa-1a957297c371 None None] MFA Rules not processed for user `a7ecfb9570f746a0b1a9c29ff7cd64ef`. Rule list: `[]` (Enabled: `True`). {{(pid=80423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:50:03.001711 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-07889667-6d69-4da6-82fa-1a957297c371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:03.122291 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1410/2864] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:50:02 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:50:03.132244 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:03.154156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09da8a3-695b-438b-9f32-3248348bf8e6 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:03.162870 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:03.164012 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:03.164380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:03.164672 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] PATH_INFO: `/v3/projects` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:03.165417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:03.167935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:03.206909 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1455/2865] 10.222.0.72 () {60 vars in 1152 bytes} [Mon Jul 11 14:50:03 2022] POST /identity/v3/projects => generated 373 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:50:03.212589 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889667-6d69-4da6-82fa-1a957297c371 None None] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:03.227815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07889667-6d69-4da6-82fa-1a957297c371 None None] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:03.329879 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:03.330632 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] REQUEST_METHOD: `POST` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:03.330953 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:03.331239 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] PATH_INFO: `/v3/users` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:03.331965 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:03.334334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:03.359203 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1411/2866] 10.222.0.72 () {60 vars in 1143 bytes} [Mon Jul 11 14:50:03 2022] POST /identity/v3/users => generated 293 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 11 14:50:03.363785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:03.376109 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70282fd4-b60d-4b34-98ad-e6032f2a063e admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:03.381196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:03.381942 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:03.382408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:03.382721 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:03.383497 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:03.385171 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:03.398531 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1456/2867] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:50:03 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 11 14:50:03.403929 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:03.415377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a230d74a-fc84-43e3-a501-138367a514f4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:03.421002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:03.421968 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:03.422244 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:03.422572 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] PATH_INFO: `/v3/projects/549b1da9bf79494ca7fd44ba94225712/users/582b415d21ec424d85f76c5e2056f2cc/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:03.441444 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=549b1da9bf79494ca7fd44ba94225712, user_id=582b415d21ec424d85f76c5e2056f2cc, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:03.444347 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:03.462816 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1412/2868] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:50:03 2022] PUT /identity/v3/projects/549b1da9bf79494ca7fd44ba94225712/users/582b415d21ec424d85f76c5e2056f2cc/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:50:03.469804 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:03.487263 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cf3b7-8add-459c-92c3-b6f1f28a92ea admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:03.555028 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:03.591460 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a22124c-64a4-4176-a05a-447ac13575ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:03.592340 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22124c-64a4-4176-a05a-447ac13575ba admin admin] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:03.592746 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22124c-64a4-4176-a05a-447ac13575ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:03.593190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22124c-64a4-4176-a05a-447ac13575ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:03.594122 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a22124c-64a4-4176-a05a-447ac13575ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:03.596958 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a22124c-64a4-4176-a05a-447ac13575ba admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:03.613132 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1457/2869] 10.222.0.72 () {58 vars in 1121 bytes} [Mon Jul 11 14:50:03 2022] GET /identity/v3/roles => generated 1420 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 11 14:50:03.619468 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:03.637098 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109241a0-f830-4c47-8515-5a56736d82cf admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:03.644449 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:03.645606 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:03.646102 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:03.646598 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] PATH_INFO: `/v3/projects/549b1da9bf79494ca7fd44ba94225712/users/582b415d21ec424d85f76c5e2056f2cc/roles/7ce0e1d27bc5490c885bee2563cbd603` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:03.650338 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=549b1da9bf79494ca7fd44ba94225712, user_id=582b415d21ec424d85f76c5e2056f2cc, role_id=7ce0e1d27bc5490c885bee2563cbd603)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:03.653923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:03.662333 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:03.684839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1413/2870] 10.222.0.72 () {60 vars in 1482 bytes} [Mon Jul 11 14:50:03 2022] PUT /identity/v3/projects/549b1da9bf79494ca7fd44ba94225712/users/582b415d21ec424d85f76c5e2056f2cc/roles/7ce0e1d27bc5490c885bee2563cbd603 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:50:10.569684 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9710725-b944-41b2-a3bb-165d26209b8a None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:10.570946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9710725-b944-41b2-a3bb-165d26209b8a None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:10.571506 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9710725-b944-41b2-a3bb-165d26209b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:10.637097 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-e9710725-b944-41b2-a3bb-165d26209b8a None None] MFA Rules not processed for user `582b415d21ec424d85f76c5e2056f2cc`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:50:10.641046 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-e9710725-b944-41b2-a3bb-165d26209b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:10.830129 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1458/2871] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:50:10 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 11 14:50:10.894307 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:10.898288 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:10.998334 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42eff2-7210-46bc-bacd-7ee9f216eda4 admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:11.006335 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:11.007624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:11.008017 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:11.008372 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:11.093223 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:11.095815 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:11.107473 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1414/2872] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:10 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:50:13.507069 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] REQUEST_METHOD: `POST` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:13.507872 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:13.508073 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:13.546385 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.auth.core [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] MFA Rules not processed for user `a7ecfb9570f746a0b1a9c29ff7cd64ef`. Rule list: `[]` (Enabled: `True`). {{(pid=80424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 11 14:50:13.549357 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:13.560734 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1459/2873] 10.222.0.72 () {58 vars in 957 bytes} [Mon Jul 11 14:50:13 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 11 14:50:14.351126 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:14.523479 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b899aba-cbf8-40e8-bc67-58eca91c985d service nova] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:14.530793 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:14.531825 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:14.532313 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:14.532849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:14.538075 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:14.558530 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:14.560843 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:14.575674 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1415/2874] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:14 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:50:19.624417 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:19.626855 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:19.647673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee3819-75bd-493d-b8d8-b64a21e5e89f None None] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:19.654461 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:19.655670 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:19.656370 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:19.657052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:19.660475 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:19.694146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:19.697066 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:19.721190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1460/2875] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:50:24.568908 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=84 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:25.612179 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:25.622505 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c70a2-4cd3-4aa9-8bc6-dc03a82a9dc3 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:25.626637 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:25.627319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:25.627608 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:25.627883 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:25.630390 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:25.718804 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:25.720630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:25.734419 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1416/2876] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:50:29.183099 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:29.198521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79e73e-f9f1-4301-a88b-0cd9e9f79264 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:29.204432 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:29.205569 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:29.206166 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:29.206712 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:29.216085 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:29.228767 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:29.231032 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:29.242969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1461/2877] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:29 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:50:30.134609 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:30.138643 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:30.291011 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7478f430-fa84-4d54-86aa-071a42362f65 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:30.316413 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:30.317913 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:30.318740 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:30.319487 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:30.323729 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:30.351569 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:30.355049 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:30.375283 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1417/2878] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:30 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:50:35.190181 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:35.195196 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:35.216442 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=61 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:35.221536 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dd5ad7-0607-40c9-8967-109aa556a895 service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:35.227232 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:35.228271 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:35.228642 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:35.229023 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] PATH_INFO: `/v3/users/0f8e300cd96e446db7430044e69d997a` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:35.230737 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f8e300cd96e446db7430044e69d997a)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:35.232922 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:35.311005 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f8e300cd96e446db7430044e69d997a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.323504 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f8e300cd96e446db7430044e69d997a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.325407 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f8e300cd96e446db7430044e69d997a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.336188 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f8e300cd96e446db7430044e69d997a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.353935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f8e300cd96e446db7430044e69d997a'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.363821 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1462/2879] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:50:35 2022] DELETE /identity/v3/users/0f8e300cd96e446db7430044e69d997a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:50:35.673484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:35.694647 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578009bf-a8b0-41bb-8510-8d27a8868dfd service placement] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:35.777927 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:35.779047 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:35.779427 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:35.779803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] PATH_INFO: `/v3/projects/3d541b7a72dc4b4494a42bda6fa75026` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:35.781246 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d541b7a72dc4b4494a42bda6fa75026)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:35.783885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:35.824830 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:35.852275 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d541b7a72dc4b4494a42bda6fa75026', 'request_id': 'req-b2db6496-16d7-412a-b30b-40f72916c65d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.861865 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d541b7a72dc4b4494a42bda6fa75026', 'request_id': 'req-b2db6496-16d7-412a-b30b-40f72916c65d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.873134 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d541b7a72dc4b4494a42bda6fa75026', 'request_id': 'req-b2db6496-16d7-412a-b30b-40f72916c65d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.893114 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] The token cache is being invalidate because project 3d541b7a72dc4b4494a42bda6fa75026 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:50:35.893556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.894396 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:35.896294 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1418/2880] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:50:35 2022] DELETE /identity/v3/projects/3d541b7a72dc4b4494a42bda6fa75026 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 11 14:50:35.900777 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:35.903015 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:35.919101 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a668b0-3646-43fa-ad76-65f94b80c1ac admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:35.997100 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:35.998673 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:35.999205 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:35.999730 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] PATH_INFO: `/v3/users/d652ee5c440b41c980191f5c0c1eeb8e` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:36.002086 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d652ee5c440b41c980191f5c0c1eeb8e)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:36.005692 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:36.069711 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd652ee5c440b41c980191f5c0c1eeb8e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.080193 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd652ee5c440b41c980191f5c0c1eeb8e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.080851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd652ee5c440b41c980191f5c0c1eeb8e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.091443 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd652ee5c440b41c980191f5c0c1eeb8e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.104561 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:36.108380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd652ee5c440b41c980191f5c0c1eeb8e'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.118528 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1463/2881] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:50:35 2022] DELETE /identity/v3/users/d652ee5c440b41c980191f5c0c1eeb8e => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:50:36.323839 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:36.326194 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:36.347671 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2db6496-16d7-412a-b30b-40f72916c65d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:36.421277 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:36.422185 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:36.422518 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:36.422800 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] PATH_INFO: `/v3/projects/569751527e5c4587b69f8bee9b359fc2` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:36.424152 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] RBAC: Authorizing `identity:delete_project(project_id=569751527e5c4587b69f8bee9b359fc2)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:36.426201 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:36.507125 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '569751527e5c4587b69f8bee9b359fc2', 'request_id': 'req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.518873 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '569751527e5c4587b69f8bee9b359fc2', 'request_id': 'req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.529093 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '569751527e5c4587b69f8bee9b359fc2', 'request_id': 'req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.545907 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] The token cache is being invalidate because project 569751527e5c4587b69f8bee9b359fc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:50:36.546245 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.546952 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:50:36.548889 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1419/2882] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:50:36 2022] DELETE /identity/v3/projects/569751527e5c4587b69f8bee9b359fc2 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:50:44.318468 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:44.320605 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:44.389671 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c16e85b-b4a4-4571-9142-0c59bfa22747 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:44.450134 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:44.450851 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:44.451143 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:44.451425 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:44.453515 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:44.465402 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:44.466991 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:44.478328 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1464/2883] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:44 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:50:46.455580 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:50:46.457242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:46.471586 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4741c0-1d5d-46fa-852a-a0b5fc4f896d admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:50:46.477846 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:50:46.479014 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:50:46.479508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:50:46.480002 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:50:46.483196 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:50:46.548925 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:46.587472 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:50:46.590184 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:50:46.609281 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1420/2884] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:50:46 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:50:54.472788 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:DELETE=1,keystone:SELECT=48 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:50:56.600484 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=10 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:52:58.990182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:52:59.000258 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf6e7d6-f1f0-4a23-8238-8f0db40df03f service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:52:59.004677 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:52:59.005363 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:52:59.005652 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:52:59.005935 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:52:59.008526 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:52:59.020055 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:52:59.021578 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:52:59.032255 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1465/2885] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:52:58 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:52:59.137381 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:52:59.138986 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:52:59.151006 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dafb4c-6c97-4983-8e31-61be3b55ed11 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:52:59.154964 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac5254fb5eff4108be9ac3992e05a99c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:52:59.155644 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:52:59.155930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:52:59.156208 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:52:59.158972 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:52:59.242555 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:52:59.244045 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:52:59.258031 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1421/2886] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:52:59 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:53:00.541123 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:53:00.543044 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:53:00.563890 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4196f0b-6001-4cfb-973f-28065bd15bca service neutron] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:53:00.571156 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:53:00.571955 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:53:00.572307 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:53:00.572775 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:53:00.575369 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:53:00.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:53:00.594472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:53:00.609785 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1466/2887] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:53:00 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:53:09.250923 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=34 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:53:10.602700 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=12 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:53:24.201714 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:53:24.203295 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:53:24.287181 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097f2c6-ea84-4d45-a8c1-8ef9129331e5 service neutron] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:53:24.293329 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:53:24.294020 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:53:24.294319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:53:24.294630 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:53:24.296736 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:53:24.310523 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:53:24.312237 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:53:24.324490 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1422/2888] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:53:24 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:53:34.318988 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=32 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:54:26.268170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:26.269722 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:26.289709 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88d3560-2dde-4e18-933d-fdd8dfe3d1c2 service nova] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:26.297076 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:26.297874 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:26.298230 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:26.298624 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:26.301817 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:26.316204 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:26.318380 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:26.331985 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1467/2889] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:54:26 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:54:28.677851 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:28.699319 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610644b3-8659-45b4-bcad-c4c90cdcee92 service placement] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:28.708221 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2165eacee8954dce975ba81ba8ac4232', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:28.709105 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:28.709493 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:28.709885 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:28.713066 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:28.734023 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:28.736991 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:28.758320 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1423/2890] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:54:28 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 11 14:54:35.168052 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:35.169946 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:35.188896 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7d28f-d96f-4126-af54-f938f68a4079 service nova] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:35.195237 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:35.195993 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:35.196416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:35.196818 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:35.199304 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:35.220630 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:35.224376 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:35.242629 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1468/2891] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:54:35 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:54:38.748676 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=6 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:54:39.153412 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:39.171508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa902c28-772e-43d5-a385-2c7c7a8d55df service nova] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:39.179866 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:39.180918 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] REQUEST_METHOD: `GET` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:39.181546 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:39.182068 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:39.184364 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:39.201701 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:39.203849 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:39.218046 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1424/2892] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:54:39 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:54:39.264379 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:39.265837 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:39.367242 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc411c3-e230-4ce0-9723-791e5a8ef49d service placement] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:39.372967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:39.373956 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:39.374378 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:39.374833 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] PATH_INFO: `/v3/users/a7ecfb9570f746a0b1a9c29ff7cd64ef` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:39.376408 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7ecfb9570f746a0b1a9c29ff7cd64ef)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:39.379472 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:39.443470 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7ecfb9570f746a0b1a9c29ff7cd64ef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.453065 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7ecfb9570f746a0b1a9c29ff7cd64ef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.453980 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7ecfb9570f746a0b1a9c29ff7cd64ef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.464912 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7ecfb9570f746a0b1a9c29ff7cd64ef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.479997 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7ecfb9570f746a0b1a9c29ff7cd64ef'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.487121 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1469/2893] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:54:39 2022] DELETE /identity/v3/users/a7ecfb9570f746a0b1a9c29ff7cd64ef => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:54:39.724556 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:39.743508 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f715a-34e1-4e2a-87c7-d8fdb4f91508 service placement] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:39.820995 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:39.822063 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:39.822242 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:39.822481 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] PATH_INFO: `/v3/projects/6c2327c73cc445ba97c2ef24ef45e4d5` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:39.823803 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c2327c73cc445ba97c2ef24ef45e4d5)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:39.825909 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:39.887792 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c2327c73cc445ba97c2ef24ef45e4d5', 'request_id': 'req-bb22edfe-3da5-4948-b3c7-7574d4f232cb'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.898509 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c2327c73cc445ba97c2ef24ef45e4d5', 'request_id': 'req-bb22edfe-3da5-4948-b3c7-7574d4f232cb'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.909837 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c2327c73cc445ba97c2ef24ef45e4d5', 'request_id': 'req-bb22edfe-3da5-4948-b3c7-7574d4f232cb'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.924499 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] The token cache is being invalidate because project 6c2327c73cc445ba97c2ef24ef45e4d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:54:39.924786 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.925503 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:39.927778 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1425/2894] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:54:39 2022] DELETE /identity/v3/projects/6c2327c73cc445ba97c2ef24ef45e4d5 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:54:39.932295 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:39.933979 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:39.948148 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac55743-5f10-436a-ba90-1be361b529c9 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:40.021665 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:54:40.037170 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:40.038146 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:40.038521 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:40.038967 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] PATH_INFO: `/v3/users/582b415d21ec424d85f76c5e2056f2cc` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:40.040382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] RBAC: Authorizing `identity:delete_user(user_id=582b415d21ec424d85f76c5e2056f2cc)` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:40.042416 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:40.102339 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '582b415d21ec424d85f76c5e2056f2cc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.111247 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '582b415d21ec424d85f76c5e2056f2cc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.111990 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '582b415d21ec424d85f76c5e2056f2cc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.120382 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '582b415d21ec424d85f76c5e2056f2cc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.134622 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.notifications [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '582b415d21ec424d85f76c5e2056f2cc'} {{(pid=80424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.141527 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1470/2895] 10.222.0.72 () {58 vars in 1223 bytes} [Mon Jul 11 14:54:39 2022] DELETE /identity/v3/users/582b415d21ec424d85f76c5e2056f2cc => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:54:40.362029 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Authenticating user token {{(pid=80423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:54:40.366190 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.fernet_utils [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:54:40.385930 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb22edfe-3da5-4948-b3c7-7574d4f232cb admin admin] Validating token access rules against request {{(pid=80423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:54:40.469324 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7d8b4db44114e1ba544d78dc12ef30a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e2b4fae2f8fe4791bd2046666efe9390', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:54:40.470225 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:54:40.470538 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:54:40.470821 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] PATH_INFO: `/v3/projects/549b1da9bf79494ca7fd44ba94225712` {{(pid=80423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:54:40.472146 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=549b1da9bf79494ca7fd44ba94225712)` {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:54:40.474198 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] RBAC: Authorization granted {{(pid=80423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:54:40.500675 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:54:40.532438 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '549b1da9bf79494ca7fd44ba94225712', 'request_id': 'req-3dc44752-692f-4a5f-bea7-712388c28bf3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.539377 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '549b1da9bf79494ca7fd44ba94225712', 'request_id': 'req-3dc44752-692f-4a5f-bea7-712388c28bf3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.548694 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '549b1da9bf79494ca7fd44ba94225712', 'request_id': 'req-3dc44752-692f-4a5f-bea7-712388c28bf3'} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.562624 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] The token cache is being invalidate because project 549b1da9bf79494ca7fd44ba94225712 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 11 14:54:40.562973 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.563776 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG keystone.notifications [None req-3dc44752-692f-4a5f-bea7-712388c28bf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 11 14:54:40.565633 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: [pid: 80423|app: 0|req: 1426/2896] 10.222.0.72 () {58 vars in 1232 bytes} [Mon Jul 11 14:54:40 2022] DELETE /identity/v3/projects/549b1da9bf79494ca7fd44ba94225712 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 11 14:54:50.137335 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=29,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:54:50.558687 dv-cinder-master-3-iscsi devstack@keystone.service[80423]: DEBUG dbcounter [-] [80423] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 11 14:58:31.215445 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Authenticating user token {{(pid=80424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 11 14:58:31.216964 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:58:31.385857 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b116a85-d4b8-4a35-9856-99cc260b5a28 admin admin] Validating token access rules against request {{(pid=80424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 11 14:58:31.460969 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc7f735-05c2-4f44-afd2-f9ba9d3e7b2a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5aabd82e0fd04345b63c346660e0da54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '713b51dac4f240c8b873e83e881c862f', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 11 14:58:31.461975 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7f735-05c2-4f44-afd2-f9ba9d3e7b2a service placement] REQUEST_METHOD: `GET` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 11 14:58:31.462321 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7f735-05c2-4f44-afd2-f9ba9d3e7b2a service placement] SCRIPT_NAME: `/identity` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 11 14:58:31.462772 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7f735-05c2-4f44-afd2-f9ba9d3e7b2a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 11 14:58:31.464916 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.fernet_utils [None req-2bc7f735-05c2-4f44-afd2-f9ba9d3e7b2a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 11 14:58:31.477459 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc7f735-05c2-4f44-afd2-f9ba9d3e7b2a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 11 14:58:31.479182 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc7f735-05c2-4f44-afd2-f9ba9d3e7b2a service placement] RBAC: Authorization granted {{(pid=80424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 11 14:58:31.492203 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: [pid: 80424|app: 0|req: 1471/2897] 10.222.0.72 () {60 vars in 1365 bytes} [Mon Jul 11 14:58:31 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 11 14:58:41.485761 dv-cinder-master-3-iscsi devstack@keystone.service[80424]: DEBUG dbcounter [-] [80424] Writing DB stats keystone:SELECT=89 {{(pid=80424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}